Getting Started with Kloudless Webhooks

Introduction

There are times when your application needs to keep up with changes that take place in your users’ cloud storage. Perhaps your application needs to perform an action when new files appear, or needs to update its state when files are modified/deleted. Kloudless solves this problem by making an HTTP request (a “webhook“) to a URL you define, with information on which accounts have changed. You can then query the events endpoint for more detailed information on modified files in the account.

The first release of Kloudless’ events endpoint supports Dropbox, Box, OneDrive, Evernote, Sharepoint, and OneDrive for Business. While some services like Box expose an endpoint for listing events, others such as SharePoint and OneDrive do not. Kloudless supports these services by periodically polling them for changes, allowing you to have a consistent interface to query all the services for changes.

This short walkthrough gets you up and running with Kloudless’s Webhooks. It assumes that:

  • You already have a Kloudless account and app set up (if you don’t, you can get one here)
  • You know some Python (we will be using Flask to create a basic webserver)
  • You are familiar enough with HTTP to make/understand requests
  • You can run code somewhere accessible by the Kloudless server requests

Configuring Webhooks

The webhooks for your application can be configured on the Application Details page of the developer dashboard. All you have to do is add the URL that you would like Kloudless to make requests to. Note: before the webhook is saved, Kloudless makes a test request to it to ensure that it works, so if your application isn’t up and running yet, you won’t be able to create the webhooks.

Webhooks

The Structure of the Webhooks

The requests that we make for our webhooks are extremely simple. Here is the raw HTTP of an example request Kloudless might make to the above endpoint:

POST /kloudless-webhook HTTP/1.1
Host: yourapp.example.com
Content-Length: 14
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: kloudless-webhook/1.0
X-Kloudless-Signature: VPl14EUveHQ7HcAvigiRZ7dmX7QLM5iD+pnXJv2XYok=
Content-Type: application/x-www-form-urlencoded


account=593652

As you can see it is a standard http form. We provide the id of the account that we have new events for. There will only be one account per notification. The primary purpose is to inform you that new events are available from the events endpoint. We also include a signature so that you can verify that the event actually came from our service.

Our service expects a 200 OK, with the response body consisting of only your Kloudless Application ID (available on the Developer Portal’s App Details page).

A Simple WebHooks Receiver

Using Flask we can quickly build a simple web server that can recieve these notifications, retrieve the corresponding events, and do something interesting. We start with the event receiver:

#!/usr/bin/env python
from flask import Flask, request, abort


import hmac
import base64
import hashlib
import threading


app = Flask('demo_app')
APP_ID = '<YOUR APP ID>'
API_KEY = '<YOUR API KEY>'
@app_route('/webook', methods=['POST'])
def receiver():
        # Verify the Request Signature
        digest = hmac.new(API_KEY, msg=request.data, digestmod=hashlib.sha256).digest()
        sig = base64.b64encode(digest)
        if not sig == request.headers.get('X-Kloudless-Signature'):
                abort(403)


        # Since the request is verified we can actually do something
        account_id = request.form.get('account')
        if account_id:
                threading.Thread(target=process_account, args=(account_id,)).start()
        # In order to verify that the endpoint is alive you should always return your application id with a 200 OK response
return APP_ID


if __name__ == 'main':
        app.run()

This basic structure allows you to insert just about any functionality into the process_account function. So that you don’t fetch the same events over and over again, you will have to keep track of where you are in the event stream for each account. Each time you make a request to the events endpoint you get back a cursor value that marks your current place. To track our current progress through the event stream, we are going to store the cursors for each account id in [redis])(http://redis.io). You could use your preferred data store, but redis is simple and quick to get started with. If you want to fetch new events and then process added files, write a function similar to the following (this uses python requests so that it is easier to translate to other languages):

import requests
import redis

redis_client = redis.StrictRedis(host='localhost', port=6379, db=0)
def process_account(account_id):
        # if there is an existing cursor we want to make sure we use it
        cursor = redis_client.get(account_id)
        params = {"cursor": cursor}
        headers = {"Authorization": "ApiKey %s" % API_KEY}
        url = "https://api.kloudless.com/accounts/%s/events" % account_id
response = requests.get(url, headers=headers, params=params)
if response.ok:
        data = response.json()
        events = data['objects']
        new_cursor = data['cursor']
        for event in events:
                if event['action'] == '+':  # Only care about new/modified files
                        do_something_cool(event['metadata']['id'])
        # updating cursor in datastore
        redis_client.set(account_id, new_cursor)


def something_cool(file_id):
        print "This is actually boring, there is a new file: %s" % file_id

In this case, you would put your interesting functionality in the do_something_cool function. For example, you could do some sort of document conversion or content auditing with the file id that you get. The event contains the full file metadata so you can make other decisions about what to do based on extension, parent folder, or mimetype.

Best Practices

There are a few things that should be done to ensure that your Kloudless Webhooks integration works as well as possible.

Handle WebHooks Quickly

The initial handling of the webhooks requests should be done as quickly as possible. Otherwise, a high volume of requests could easily overwhelm a naive implementation that does a large amount of work prior to sending a response to the webhook requests.

Manage Concurrency

When there are lots of changes occurring for a particular account, a large number of notifications could be sent to your server. Your application should make sure that this doesn’t cause problems. For certain types of applications, the duplicate actions would not cause any issues. However, if you have non-idempotent actions or actions that are particularly expensive in terms of computation time, then make sure there is locking, leasing, or mutual exclusion for that user for the duration of the action. This ensures that duplicate work isn’t being done while the action is running.

There are other design practices for event-driven applications, but they are beyond the scope of this post.

Conclusions

Hopefully this will have provided you with a good introduction to how to take advantage of Kloudless Webhooks so that you can build more effective event-driven applications around your user’s cloud storage. Please email support@kloudless.com or chat with us on Stack Overflow if you have any feedback, questions, or insights about building your application with Kloudless Webhooks.

How to Integrate SharePoint Without Devoted Resources

We’re excited to announce that the Kloudless API now supports SharePoint, to offer an easier way to build apps for Microsoft’s enterprise collaboration tool. With our single, unified API, developers can now integrate SharePoint along with 11 other cloud storage services.

Screenshot 2014-10-14 14

Microsoft SharePoint is still a market leader in content collaboration and has a strong hold on our developers’ customers. Since the API has historically been difficult to work with, many have de-prioritized integrations with SharePoint despite having customer demand. Our mission is to make it a no-brainer for developers to address customer requirements and give them access to a wealth of features that will be beneficial throughout the lifespan of their apps. — Eliot Sun, Kloudless CEO and Co-founder

By leveraging SharePoint through the Kloudless API, developers can offer their customers access to their enterprise data and a variety of value-add features. Integrating with SharePoint enables developers to give their customers access to a service that they already rely on to get work done. Rather than building a suite of storage and file related features themselves, developers can simply tap into the functionality available via the Kloudless API.

Several customers for SharePoint integration are already lined up, including Cotap, Taptera and PlanGrid. Key features of the new SharePoint support include search, access controls and recent files. Event notifications will be available in the coming weeks.

Kloudless’ SharePoint integration gives Taptera a great opportunity to serve our SharePoint customers without needing to devote our resources to behind-the-scenes integration work. It lets us focus on what we do best: adding great features and providing the best experience possible for our users. It’s a win for everyone. — Ian Fisher, Taptera CTO

Pricing and Availability

Developers can immediately access Kloudless support for Microsoft SharePoint through the Kloudless API, along with its supporting documentation. Pricing is based on the level of feature support and number of user accounts. The free plan supports up to 50 user accounts. Premium plans start at $19/month. Enterprise pricing is also available. Get additional details or sign up for a developer account today.

How To Download Usage Reports

You can now download API Request and Bandwidth usage reports from your Developer Portal’s Account page.

To download your reports, sign in to your Developer Portal, navigate to Account (found in the drop down under your user name). At the bottom of the page, you have the option to export your usage report.

Usage Reports Screenshot

Usage reports are generated on a calendar month basis, so you can enter the year and the month for your report. Each report gives you an aggregated view for all of your applications, with each entry providing you with request-specific information, including which account(s) were involved in the call as well as bandwidth used by each request. Use these reports to understand your users’ behavior around their file interactions, specifically as it relates to your app.

If you have any questions, comments or feedback about usage reports, let us know by sending us an email. Don’t have Kloudless yet? Sign up for your free developer account and check out the open-sourced SDKs and UI elements on GitHub.

Now Open-Sourced: Kloudless File Explorer

Kloudless has open-sourced its File Explorer, part of the Kloudless UI toolkit. The File Explorer allows users to browse and select files and folders from their storage services.

Developers can immediately access File Explorer on GitHub, along with other software development kits (SDKs) and UI tools.

Open-sourcing File Explorer enables easier customizations, debugging and self-hosting for developers. Developers can now ensure consistent branding, smooth user experience and and native-look and feel.

fileexplorer

Check out the jsFiddle example of File Explorer and use Kloudless to quickly integrate cloud storage services into your app.

https://www.flickr.com/photos/neospire/3595638270/in/photolist-ajpfMK-dZ4QZP-6tJAau-dZaCMA-j1yjj3-6tJA2d-ajpfNM-dZ4wtK-6Ta4NB-6tJzZh-6Ta4Qg-dZ5xUX-6fhifq-8Lzsie-6tEshe-dZb25U-ajs4aC-6tJAa7-jDgBDx-6tJzZS-ajs4eY-i6RYuf-5nSr4j-7MWjax-avnocW-bLXQ9c-fh3dY7-6tJA47-5scYHD-dZ4Fsx-6MhayK-4uZrYi-dZ8qHy-app8hm-38FPNk-oj1Gq-oiiLU-5szmui-6tEsgr-dZatkC-fhep2Q-ooXjK6-b3wMk-4v4vG5-oiiJx-6tEs88-c2Gpf-6tJA3W-6tEs92-6tEs9r

Loathe Latency? Updated JS Libraries and New Regions

Updated JS library URLs

We’ve switched to using a CDN to serve Kloudless JS libraries. While the old URLs work, you’re encouraged to switch to the new URLs for the File Explorer and Authenticator JS libraries.

Since a CDN optimizes for serving static files closer to your user, switching to the new URLs for either or both JS libraries will result in less latency to load the JS file. Reducing the bottleneck will give your users a better experience with the File Explorer and Authenticator.

New Regions Added

Kloudless has also added the us-east-1 AWS region in addition to us-west-2, for faster API requests as well as quicker downloads and uploads for your end users who are closer to the east coast. Sign up for your free Kloudless developer account to experience the speed yourself!

https://www.flickr.com/photos/neospire/3595638270/in/photolist-ajpfMK-dZ4QZP-6tJAau-dZaCMA-j1yjj3-6tJA2d-ajpfNM-dZ4wtK-6Ta4NB-6tJzZh-6Ta4Qg-dZ5xUX-6fhifq-8Lzsie-6tEshe-dZb25U-ajs4aC-6tJAa7-jDgBDx-6tJzZS-ajs4eY-i6RYuf-5nSr4j-7MWjax-avnocW-bLXQ9c-fh3dY7-6tJA47-5scYHD-dZ4Fsx-6MhayK-4uZrYi-dZ8qHy-app8hm-38FPNk-oj1Gq-oiiLU-5szmui-6tEsgr-dZatkC-fhep2Q-ooXjK6-b3wMk-4v4vG5-oiiJx-6tEs88-c2Gpf-6tJA3W-6tEs92-6tEs9r

We’d love to hear your thoughts on Kloudless — reach out on Twitter @KloudlessAPI, StackOverflow, IRC, or in the comments below.

File Explorer Updates: Two New Ways to Access Files

Downloading files via File Explorer links

The File Explorer now accepts an option named ‘direct_link’. When true, the links created via the File Explorer will download the file for the user, rather than present a view for the file in the appropriate cloud storage service.

This functionality enables your app’s users to download and create local copies of the file — perfect for collaboration and file transfer use-cases.

Embedding links in web pages

Speaking of direct links, they can now be accessed with an “inline=true” query parameter that sets the Content-Disposition of the downloaded file to “inline” rather than “attachment”.

This is useful when embedding links in web pages, since you can display the content for your users directly, instead of through the cloud storage service.

Give Kloudless a try — we’re available to help on Twitter @KloudlessAPI, StackOverflow, IRC, or in the comments below.

1947 partnership

Working with The 1947 Partition Archive to Record Oral Histories

At Kloudless, it’s awesome seeing all the cool stuff being built on our API. Once in a while, we get the opportunity to work with a nonprofit organization with an amazing philanthropic mission.

Today, I’m thrilled to announce that we’ve partnered with The 1947 Partition Archive to make recording oral histories easier for everyone around the world.

1947 partnership

The 1947 Partition Archive is a crowdsourced collection of oral histories documenting the Partition of British India, the largest mass refugee crisis in the history of the world. The organization provides a platform to “collect, archive and display oral histories that document not only Partition, but pre-Partition life and culture as well as post-Partition migrations and life changes.”

The Kloudless team is excited to help in this effort by making it easy for oral history collectors around the world to contribute their stories to the Archive. The Kloudless gives the Archive a user-friendly tool to accept video files of any size and resume paused uploads, which is especially important in areas with unreliable internet connections.

At Kloudless, we believe that those doing good have enough on their plates and shouldn’t have to deal with the complexities of file uploads or any other file/storage features. Nonprofit organizations committed to charitable missions are eligible for heavily discounted or completely free service!

Contact us at hello@kloudless.com to learn more about how we work with nonprofits.

The Emergence of Storage API’s & File App Ecosystems

We’re excited to announce that Kloudless co-founder and VP of Engineering Vinod Chandru will be speaking at DataWeek + API World 2014 Conference & Expo (Sept 15-17), San Francisco’s largest Data + API conference of 2014!

DataWeek + API World 2014 Conference & Expo has more than 100+ talks and opportunities to interact with 200+ new data & API technologies, including Kloudless.

vinod1

Come see our panel:
The Emergence of Storage API’s & File App Ecosystems
Tues., Sept. 16 @ 2pm

We’re offering our community a free OPEN pass to the event! Your OPEN pass will get you into:

  • All OPEN talks across the DataWeek conference (see schedule)
  • The API World conference (see schedule)
  • The DataWeek + API World Expo with 35+ exhibitors
  • The Data + API Hackathon (Sept 13-14)
  • All week-long partner events!

Get your free OPEN pass here by August 30th

Get 25% off the PRO pass for access to all talks and workshops

Come join us at DataWeek + API World, along with speakers from Google, IBM, Linkedin, The Economist, ReadWrite, HP, Dun & Bradstreet, Leap Motion, Visual.ly, Oauth.io, and hundreds more covering topics across Big Data, Data Science-as-a-Service, API Design, Data Visualization, Connected Cars, and the Internet of Things.

Can’t wait to see you there!