Office 365 PowerShell queries via REST: Maximizing the Kloudless Pass-through API

In our previous post, we announced the availability of the Kloudless Pass-through API. The Pass-through API enables your application to make API requests directly to third-party services, while still using Kloudless’s unified APIs. In this blog post, we’ll discuss how to access the Office 365 PowerShell via the Kloudless REST API to perform administrative tasks in Office 365.

Building with our easy-to-use REST API offers many benefits. We handle the complexities of integrating with each service behind the scenes so you  don’t have to. This speeds up your integration time and decreases future maintenance. We’ve extended the same principle to our Pass-through API by introducing special capabilities such as enabling PowerShell queries for Office 365 admin accounts.

Office 365 PowerShell provides several remote management commands that can be used to administer your Office 365 tenant, similar to how you would via the Office 365 admin center web application. For example, a broad set of security and compliance features can be accessed via theSecurity & Compliance Center cmdlets by connecting to Office 365 using remote PowerShell. Normally, you would need access to a PowerShell prompt in order to access this functionality. The Kloudless REST API handles the heavy-lifting and enables you to access this functionality via our REST API.

Invoking Office 365 Security Center cmdlets via the Kloudless Pass-through API

To begin, connect a SharePoint Online admin account to your Kloudless application. The easiest way to do this is by logging into your Kloudless account and then navigating to the Interactive Docs. Click the “Add Account” button and then click on SharePoint Online under the “Admin accounts” section towards the bottom of the pop-up that opens.

idjg03n

Once you’ve connected your account, you will receive a Kloudless Account ID that can be used for API requests to the Kloudless API. You are now ready to make Pass-through API requests to this admin Office 365 account!

While SharePoint Online REST API requests can be performed without any additional configuration, the PowerShell queries described in this blog post are available in Kloudless Enterprise and also require special permission to access by Kloudless Enterprise developers. Please contact us at support@kloudless.com to learn how to enable this capability for your Kloudless Enterprise instances.

Request Format

The format of PowerShell pass-through API requests is as follows:

URL: https://api.kloudless.com/v1/accounts/{account_id}/raw
  • {account_id} is the Kloudless account ID of the SharePoint Online admin account connected.
Headers (described in the Pass-through API docs):
  • X-Kloudless-Raw-URI: http://powershell/ This special value indicates the request should be translated to a PowerShell query.
  • X-Kloudless-Raw-Method: POST
  • Authorization: Bearer {account_bearer_token} OR Authorization: APIKey {application_api_key} See our Authentication Docs for more information on authorizing API requests.
Body
JSON data in the format below:

    {
      "category": "o365-security",
      "command": {cmdlet_name},
      "options": {
        ... option name: value mappings if required ...
      }
    }
At the current time, only Office 365 Security and Compliance Center cmdlets ("category": "o365-security") and Exchange Online cmdlets ("category": "exchange") are available via the Kloudless API. If you would like access to other remote PowerShell cmdlets, please contact us at support@kloudless.com.

Examples of Requests

An example of a curl request with the format described above would be:

curl -H "Authorization: APIKey {api_key}" \
     -H "X-Kloudless-Raw-URI: http://powershell/ \
    https://api.kloudless.com/v1/accounts/{account_id}/raw \
    --data '{body}'

Please replace the {api_key}, {account_id} and {body} values with your API Key, connected account’s ID and JSON data for PowerShell respectively.

Here are some examples of Body data to use in {body} for specific cmdlets:

Get-ComplianceCase
Obtaining a list of compliance cases.

{
  "category": "o365-security",
  "command": "Get-ComplianceCase"
}
An example of a curl request for this would be:

curl -H "Authorization: APIKey 123ABC" \
     -H "X-Kloudless-Raw-URI: http://powershell/" \
     https://api.kloudless.com/v1/accounts/123/raw \
     --data '{"category": "o365-security", "command": "Get-ComplianceCase"}'
New-ComplianceCase
Create a new compliance case.

{
  "category": "o365-security",
  "command": "New-ComplianceCase",
  "options": {
    "Name": "test new case 2",
    "Description": "This case is created via curl"
  }
}
An example curl request would be identical to the one used for Get-ComplianceCase but with the new value above for --data instead.
New-CaseHoldPolicy
Create a new hold policy for a case.

{
  "category": "o365-security",
  "command": "New-CaseHoldPolicy",
  "options": {
    "Case": "3b4de8d5-13cb-4291-bdd0-b6e2bb82a08e",
    "Name": "New Hold",
    "SharePointLocation": "https://kloudless.sharepoint.com/test subsite/"
  }
}
where "3b4de8d5-13cb-4291-bdd0-b6e2bb82a08e" is the Identity GUID of the Case to add the legal hold policy to. This corresponds to the Locations section of a Hold when editing a Case’s Holds at https://protection.office.com/#/ediscovery.
New-CaseHoldRule
Create a rule to add to a hold policy.

{
  "category": "o365-security",
  "command": "New-CaseHoldRule",
  "options": {
    "Policy": "New Hold",
    "Name": "New Rule",
    "ContentMatchQuery": "SSN"
  }
}
where "New Hold" is the name of the hold I created previously. This corresponds to the Conditions section of a Hold.

Similarly, other policies such as retention policies can also be created, and existing objects can be deleted:

New-RetentionCompliancePolicy
Create a new preservation policy.

{
  "category": "o365-security",
  "command": "New-RetentionCompliancePolicy",
  "options": {
    "Name": "Test new policy",
    "SharePointLocation": "https://kloudless.sharepoint.com/"
  }
}
This creates a policy but has not yet created a rule to add to it, which can be done with .
Remove-ComplianceCase
Deleting a compliance case.

{
  "category": "o365-security",
  "command": "Remove-ComplianceCase",
  "options": {
    "Identity": "94b99324-5574-4220-b081-1b689cb386af",
    "Confirm:$false": null
  }
}
where "94b99324-5574-4220-b081-1b689cb386af" is the Identity GUID of the Case to remove.

Future capabilities of the Pass-through API

The Pass-through API provides provides a powerful new way to access third-party features via Kloudless as shown above. We’re excited to make this available to all developers on our platform and would love to hear any feedback or suggestions in our developer forum.

Getting Started with Kloudless Webhooks

Introduction

There are times when your application needs to keep up with changes that take place in your users’ cloud storage. Perhaps your application needs to perform an action when new files appear, or needs to update its state when files are modified/deleted. Kloudless solves this problem by making an HTTP request (a “webhook“) to a URL you define, with information on which accounts have changed. You can then query the events endpoint for more detailed information on modified files in the account.

The first release of Kloudless’ events endpoint supports Dropbox, Box, OneDrive, Evernote, Sharepoint, and OneDrive for Business. While some services like Box expose an endpoint for listing events, others such as SharePoint and OneDrive do not. Kloudless supports these services by periodically polling them for changes, allowing you to have a consistent interface to query all the services for changes.

This short walkthrough gets you up and running with Kloudless’s Webhooks. It assumes that:

  • You already have a Kloudless account and app set up (if you don’t, you can get one here)
  • You know some Python (we will be using Flask to create a basic webserver)
  • You are familiar enough with HTTP to make/understand requests
  • You can run code somewhere accessible by the Kloudless server requests

Configuring Webhooks

The webhooks for your application can be configured on the Application Details page of the developer dashboard. All you have to do is add the URL that you would like Kloudless to make requests to. Note: before the webhook is saved, Kloudless makes a test request to it to ensure that it works, so if your application isn’t up and running yet, you won’t be able to create the webhooks.

Webhooks

The Structure of the Webhooks

The requests that we make for our webhooks are extremely simple. Here is the raw HTTP of an example request Kloudless might make to the above endpoint:

POST /kloudless-webhook HTTP/1.1
Host: yourapp.example.com
Content-Length: 14
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: kloudless-webhook/1.0
X-Kloudless-Signature: VPl14EUveHQ7HcAvigiRZ7dmX7QLM5iD+pnXJv2XYok=
Content-Type: application/x-www-form-urlencoded

account=593652

As you can see it is a standard http form. We provide the id of the account that we have new events for. There will only be one account per notification. The primary purpose is to inform you that new events are available from the events endpoint. We also include a signature so that you can verify that the event actually came from our service.

Our service expects a 200 OK, with the response body consisting of only your Kloudless Application ID (available on the Developer Portal’s App Details page).

A Simple WebHooks Receiver

Using Flask we can quickly build a simple web server that can receive these notifications, retrieve the corresponding events, and do something interesting. For an extremely simple server you can use for testing purposes, check out this Gist.

Here, we will describe a more useful receiver that verifies the webhook sender and launches a task. We start with the event receiver:

#!/usr/bin/env python
from flask import Flask, request, abort

import hmac
import base64
import hashlib
import threading

app = Flask('demo_app')
APP_ID = ''
API_KEY = ''
@app.route('/webhook', methods=['POST'])
def receiver():
    # Verify the Request Signature
    digest = hmac.new(API_KEY, msg=request.data, 
                      digestmod=hashlib.sha256).digest()
    sig = base64.b64encode(digest)
    if not sig == request.headers.get('X-Kloudless-Signature'):
        abort(403)

    # Since the request is verified we can actually do something
    account_id = request.form.get('account')
    if account_id:
        threading.Thread(target=process_account, args=(account_id,)).start()

    # In order to verify that the endpoint is alive you should always return
    # your application id with a 200 OK response
    return APP_ID

if __name__ == 'main':
    app.run()

This basic structure allows you to insert just about any functionality into the process_account function. So that you don’t fetch the same events over and over again, you will have to keep track of where you are in the event stream for each account. Each time you make a request to the events endpoint you get back a cursor value that marks your current place. To track our current progress through the event stream, we are going to store the cursors for each account id in [redis])(http://redis.io). You could use your preferred data store, but redis is simple and quick to get started with. If you want to fetch new events and then process added files, write a function similar to the following (this uses python requests so that it is easier to translate to other languages):

import requests
import redis

redis_client = redis.StrictRedis(host='localhost', port=6379, db=0)

def process_account(account_id):
    # if there is an existing cursor we want to make sure we use it
    cursor = redis_client.get(account_id)
    params = {"cursor": cursor}
    headers = {"Authorization": "ApiKey %s" % API_KEY}
    url = "https://api.kloudless.com/v1/accounts/%s/events" % account_id
    response = requests.get(url, headers=headers, params=params)

    if response.ok:
        data = response.json()
        events = data['objects']
        new_cursor = data['cursor']
        for event in events:
            if event['type'] == 'add': # Only care about new/modified files
                do_something_cool(event['metadata']['id'])
        # updating cursor in datastore
        redis_client.set(account_id, new_cursor)

def do_something_cool(file_id):
    print "There is a new file: %s" % file_id

In this case, you would put your interesting functionality in the do_something_cool function. For example, you could do some sort of document conversion or content auditing with the file id that you get. The event contains the full file metadata so you can make other decisions about what to do based on extension, parent folder, or mime-type.

Best Practices

There are a few things that should be done to ensure that your Kloudless Webhooks integration works as well as possible.

Handle WebHooks Quickly

The initial handling of the webhooks requests should be done as quickly as possible. Otherwise, a high volume of requests could easily overwhelm a naive implementation that does a large amount of work prior to sending a response to the webhook requests.

Manage Concurrency

When there are lots of changes occurring for a particular account, a large number of notifications could be sent to your server. Your application should make sure that this doesn’t cause problems. For certain types of applications, the duplicate actions would not cause any issues. However, if you have non-idempotent actions or actions that are particularly expensive in terms of computation time, then make sure there is locking, leasing, or mutual exclusion for that user for the duration of the action. This ensures that duplicate work isn’t being done while the action is running.

There are other design practices for event-driven applications, but they are beyond the scope of this post.

Conclusions

Hopefully this will have provided you with a good introduction to how to take advantage of Kloudless Webhooks so that you can build more effective event-driven applications around your user’s cloud storage. Please email support@kloudless.com or chat with us on Stack Overflow if you have any feedback, questions, or insights about building your application with Kloudless Webhooks.

How to create folders using the File Explorer

Sometimes your users might have documents that don’t fit into their current organizational scheme. They’ll have to create a new folder.

Going to whatever cloud storage service of their choice, creating a new folder, and then going back to your app is certainly an option, but it’s a painful one. 😦 Why put your users through that?

Create a smoother user experience and increase the time your users spend in your app by using folder creation. Folder creation is available when you implement the File Explorer. Your app’s users can now select new locations in their cloud storage when your app prompts them to choose a folder.

 

Here’s what that experience looks like for your end users. First, they’ll click on the folder icon to create a new folder:

Create New Folder One

 

Then they’ll see this screen:

Screenshot 2014-08-12 12.17.16

 

They just need to fill out the form field with the name of the folder what they’d like to use. I chose “Folder Rename Demo” (very exciting, I know!):

Screenshot 2014-08-12 12.23.08

 

Clicking save creates and saves the folder within their desired cloud storage service:

Screenshot 2014-08-15 14.44.34

 

There you have it — a perfectly pristine folder that your users can create from within your app. You can learn more about the file explorer here or try a demo.

As always, please leave us a comment below, email us, open an issue in the relevant GitHub repository, reach out to us on Twitter @KloudlessAPI, or talk with us on IRC on FreeNode in #kloudless if you need anything!

Send Documents on Feature Phones with Twilio and Kloudless

Hey hey! Following an intense month of events, I got inspired by all the cool hacks at Hack(bright) for Good, companies at TechCrunch Disrupt and developers at Kloud Kantina.

I thought txt.us, a WhatsApp for feature phones (find their project source code here), was especially cool, so I wanted to see if I could take the idea one step further: the ability to send files via SMS using a link.

txt_us_hackbrightforgood_2014

Team txt.us presenting at #hackbrightforgood

[tl;dr check out the finished Twilio + Kloudless project and the source code on GitHub to see an app that lets you send files stored in the cloud via SMS]

The Code

The application provides the following commands, which are sent via SMS to the application’s Twilio phone number:

  • ls – returns indexed list of visible folders
  • cd (<index>|..) – change to listed directory or to parent respectively
  • get <index> – Get an SMS with a link to download the file
  • send <index> <phone number> – send an SMS with link to file to number (e.g. +15555555555)
  • reset – starts your session back from the beginning

It is built using the Flask application that you created by the end of the Twilio SMS and MMS Python Quickstart. I had to add a bit of extra code to make it useable as a public web application, but the main cloud storage code interaction code is in the handler.py file. There I made the following translations from what I wanted into the Kloudless Python SDK code:

  • ls for accounts: kloudless.Account.all()
    this will list all accounts that your application has access to

    • In the final product, I end up storing account ID’s with phone numbers so that not every phone number sees every account.
    • To look up an individual account, you can simply do kloudless.Account.retrieve(account_id) where account_id is the id number of the account you want.
  • ls for a directory, if you have the right account object in account and the folder id in folder_id: account.folders(id=folder_id).contents()
    • To list the contents of the root folder for a particular account (with the account object stored in account): account.folders().contents() this works because the id defaults to the root directory of the cloud storage account.
  • get if you have the file object stored in file_obj and the account in account: account.links.create(file_id=file.id)
    • This returns a link object containing a url that the user can visit to preview (if the source supports it) or download the file.
    • I decided to just return a link, since I wasn’t sure whether all file types could be sent via MMS and most phones have some kind of web browser (though it might be fairly bare bones)
  • send is basically like get except that you are sending it to someone else so you have to make an extra Twilio API call

Once the basic commands were mapped out, it was relatively simple to wrap the functionality in an object that we can easily re-use.

The Twilio functionality I used was also very straight forward. For handling web hook responses, I simply create a TWiML response like so:

resp = twilio.twiml.Response()
resp.message("This should be the response from the command handler")

This can be returned to the Twilio webhook to send a response to an incoming text message. The command handler I made just outputs the message that we want to send back.

The only other functionality that I used was sending messages, which is also really straight forward:

twilio_client = TwilioRestClient(account="blah", token="blah")
twilio_client.messages.create(to="+15555555555",
                              _from="+YOURAPPNUMBER",
                              body="This is a message!")

And that is it, those are the basic pieces I used to make this app, the core parts of this application is only about 150 lines of code, the rest is just fluff to make the application more web ready (like having an actual website to log in and manage connected accounts).

Using Twilio + Kloudless

To use Twilio+Kloudless, first click the link and enter your phone number.
Check your phone for the confirmation code and enter it into the form field. Once your account is confirmed, you can connect your different cloud storage accounts via the web interface.

From your phone, text “ls” to the stated number, which then prompts you through interacting with your connected accounts. I won’t spoil all of the fun, so I’ll let you check out the project yourself.

Conclusion

Throughout this project, I didn’t have to write a single line of cloud storage service specific code, and with just those simple commands detailed above, this app works with accounts from any of the nine services supported by Kloudless.

With Twilio and Kloudless, you can use a feature phone to send files (no need for a smartphone!). Check out the code on GitHub — feel free to fork it and build your own functionality on top. If you want to get started on your own project, sign up for Twilio or Kloudless here. To bounce ideas around, leave a comment below or drop us a line at hello@kloudless.com.