Kloudless Treehouse Adventure

Eight weeks previous, the Kloudless HQ team found themselves in a literal treehouse in Guerneville, California, for their annual retreat.

Image uploaded from iOS

The treehouse in question…

Image uploaded from iOS copy

…and the accompanying view!

The following is a vivified recount of the trip’s exciting festivities—and, for all of our readers, a fascinating peek behind the scenes into Kloud culture.

Day 1

Upon entering the treehouse, VP Engineering Tim, Kloudless’s resident man of culture, was greeted with one of his favorite things as a surprise birthday present from the entire team!

IMG_6848

Tim and his beloved Johnnie Walker Blue Label Scotch Whisky

As soon as the whisky was safely delivered into Tim’s outstretched hands, beds were claimed, and the house was settled into, the team prepared to launch themselves into a bold exploration of the wild naturescapes and mysterious shorelines of Northern California…

First, our engineering team went to the beach to immerse themselves in the salty sea breeze and trademark chilly temperature characteristic of beaches of the area:

IMG_6830

The elusive “outside,” the likes of which engineers are rarely able to witness.

20170815_152259

Some of our engineering team looking fresh! I mean, just check out those shades on David “dthorman” Thorman, our Head of DevOps.

IMG_6841

CTO Vinod, walking softly while also carrying a big stick.

Once the team satisfied themselves with their amount of exposure to the outside world, they settled back into their lodgings for the night, ending the day with food, drinks, and – naturally – a couple of intense games of Catan.

The centerpiece of the night? A delicious homemade paella cooked by Dthorman and Tim:

Image uploaded from iOS (1)

Fun fact: Tim is allergic to eggs, which is probably why he made paella instead of fried rice.

Soon enough, the Kloudless team found themselves falling into a groggy paella-and-whisky-induced slumber…  


Day 2

The next day of the retreat began with the team scarfing down a plentiful breakfast of fresh blueberries, croissants, pastries, scrambled and fried eggs (which Tim was unable to consume), bacon, and orange juice.

DCIM100GOPROGOPR2143.JPG

We have no shame in admitting that we got that box of croissants in the corner from Safeway.

The goal for this carboload-fest: absorb enough energy to sustain the body for the 4-hour-long kayaking trip down the Russian River that was scheduled in the itinerary for the day.

DCIM100GOPROGOPR2145.JPG

We weren’t sure why the water was green.

DCIM100GOPROGOPR2148.JPG

CEO Eliot (in center) with the rest of the team! Fun fact: when Eliot was in college, he was a star athlete on the Cal Dragonboat Team…which of course meant that everybody wanted to be paddled around in Eliot’s kayak.

 

20170816_140622

Our lovely and chipper office manager Sophie and the effortlessly cool (though distinctly less chipper) Dthorman take a short break from rowing.

20170816_140630.png

Vinod takes a break (and a selfie) while the rest of the team continues to row the kayak along the river. Fret not – no enmity arose in response to his break, even if his lack of help did make the boat harder to row.

For dinner, Sophie played head chef as she barked out orders to Tim (acting sous chef) to mince garlic and clean chicken legs. The finished result? An incredibly scrumptious and authentic dinner of fragrant Hainan chicken rice, with a comforting yet satisfying side of fishball and Napa cabbage soup:

20170816_183405

If you can’t tell by now, food is a huge part of Kloud culture. You won’t find any Soylent bottles here.

Many thanks to Sophie’s dad for the recipe – the Kloudless team had a lot of fun making (and eating it)!

After dinner festivities included classic video games (Mario Kart and Super Smash Bros. Brawl), more board games (Codenames and more Catan), more drinks (Tim’s birthday present, as well as varieties of beer), and to top the night off, several rounds of poker.

20170816_204649

Our software engineering intern Ellen “smashed” everyone at Smash, but Eliot cruised effortlessly into first place every single time at Mario Kart.

20170817_010850

Software Engineering Intern Matthew Soh showing off his winnings for the night. (He cashed out at $8.50 with a high-roller buy in of $1 – at Kloudless, we endorse responsible gambling habits.)

All in all, a great retreat in the books for the Kloudless team! 


Join us!

Are you a board game or whiskey aficionado? Think you can beat Eliot at MarioKart, and Matthew at heads up poker?

Or do you simply want to enjoy some delicious paella, Hainan chicken rice, and ice cream with us? 

We’re hiring for engineering, marketing, and sales in our Berkeley and Taiwan offices! Find our job openings here.

Metrics and Monitoring at Kloudless

This post was written by our engineering intern, Matthew Soh.

Metrics and monitoring are important for any service. They provide critical information needed to detect and respond to incidents and issues. Kloudless deals with millions of requests, and keeping track of everything can be difficult. Kloudless has recently integrated a new metrics system to tackle this challenge.

Metrics from Kloudless can be sent to the new analytics platform for collection, analysis, and alerting. The analytics integration is available to all operators of Kloudless systems: both our own DevOps team administering our cloud version as well as operators of self-hosted Kloudless Enterprise appliances.

Usage

Let’s walk through a simple use case. Let’s say that your application uses the Kloudless Storage API and is failing to store files uploaded to your service. The potential issue could be anywhere in the stack. With the new metrics system, dashboards that display request metrics are easily accessible. Now you can quickly isolate the issue based on these metrics.

blog-post-dashboard

Chronograf dashboard with graphs of Kloudless metrics.

Here we have a simple dashboard. Core health checks to the Kloudless appliance appear to be fine. However, there is a sharp spike in the graph of Request Failures! This graph shows failures for outbound requests to upstream services. Hovering over the graph, we can look at the tags on each data series and see that there’s been a large increase in 500 errors to Box. This suggests that the issue is likely with the upstream service rather than the Kloudless API or the appliance itself. Knowing this allows us to narrow down which logs we need to look at to learn more about the error and take further steps to assess the root cause.

Request status is the mere tip of the iceberg when it comes to metrics provided by the Kloudless appliance. For a more detailed reference of available metrics, please refer to the Kloudless Enterprise Configuration guide.

How it works

The dashboard used above is built with Chronograf. It is one part of the metrics system deployed at Kloudless that uses the TICK stack, by InfluxData. TICK is comprised of Telegraf (collector), InfluxDB (datastore), Chronograf  (visualization), and Kapacitor (monitoring).

blog-post-influx

The TICK Stack. © 2017 InfluxData, Inc.

The metrics processing chain begins with Telegraf, the metrics collection daemon. Telegraf is designed to aggregate data from different sources and send them to various datastores. Sources include sysstat (a system information tool) and statsd (a common metrics daemon). The default datastore is InfluxDB, though others such as Graphite and CloudWatch can also be used. If required, the Telegraf output in the Kloudless appliance can be modified, enabling existing metrics collection or storage infrastructure to be used instead of InfluxDB.

The metrics then proceed to InfluxDB, which is designed for storing time-series data. This means that it has some neat features such as simple data retention policies and continuous queries. Data retention policies allow for time limits on metrics to expire old data. Continuous queries run at regular intervals on InfluxDB to summarise detailed data into broad overviews. For example, summations over counts of API requests are used to build daily summaries. Together, these features allow InfluxDB resource usage to be managed effectively.

Once the data is stored, Chronograf and Kapacitor work in tandem to help understand the collected metrics. Chronograf enables dashboard visualizations like the one described above to be built, while Kapacitor provides automatic monitoring so that there is no need to stare at the dashboard all day. Kapacitor’s monitoring and alerts are managed through TICKScripts which can be configured using either the Kapacitor command line client or Chronograf. We have provided sample TICKScripts which cover some common use cases.

Why we moved to the TICK stack

As the Kloudless Platform has grown, the volume and complexity of metrics data has grown with it. Previously, we were using with StatsD as a collector and Graphite to visualize metrics. This simple solution was easy to work with, but our needs have changed. Here are some of the unsupported scenarios we encountered:

  • StatsD doesn’t support tags. Tags are useful for providing context for the measurement, such as status or type of a request. Our workaround was to append tags onto the measurement name, similar to the tags used by DataDog’s DogStatsD. The downside of this approach was messy measurement names that were difficult to query.
  • Each StatsD measurement only has one value. It is sometimes useful to have a tuple of data grouped together in a single metric, or associate data such as application IDs to a metric. This isn’t possible with the StatsD+Graphite solution either.
  • The language used to query Graphite is limited to nesting of functions, and performing aggregations can be slow since the metrics are stored in flat files. This results in slow queries across multiple metrics and prevents easy use of complex operations.

These factors led us to look for a better alternative. InfluxDB seemed promising as its design was tailored for high volume time-series data and metrics collection. InfluxDB is built around measurements, which are in turn made up of many data points. A data point can have one or more values associated with it, and as many tags as needed. This addressed our first two issues right away and allowed for better querying.

For example, let’s try to determine which Kloudless applications were associated with failed API requests in the past day. We can do this with the following query:

select status, path, application_id from request_metrics_api_requests 
where time > now() - 24h and status=~/[^2][0-9]{2}/

In just one query, we’ve selected multiple values (status, path, application_id), filtered by tag values (status is not a 2XX code) and with a time frame limit (last 24 hours). This would have been messier and more tedious with our previous metrics system which did not allow associated metadata to be recorded. Additionally, InfluxDB’s time-series database design allows for tag-based queries to execute more efficiently since all tags are indexed.

Next Steps

We’ve been using the new metrics system at Kloudless to better understand, measure and monitor the performance of our hosted cloud platform. We’ve found that it has saved us time and effort when triaging issues, and we think you’ll find it useful as well. We’re excited to make this metrics system available to our enterprise customers using the Kloudless appliance. Customers using our cloud platform will also see analytics data for their Kloudless applications exposed in the developer portal in the upcoming months. As always, we would love to hear your thoughts and feedback on Twitter, comments below, or at hello@kloudless.com.

Calendar API: Availability Endpoint now… Available!

This post was written by our software engineer, Ryan Connor.

Finding an appropriate meeting time can take a lot of effort, even with just a few participants who have only a single calendar. Finding a meeting time that is sure to work for many participants with multiple calendars managed by several cloud services? Good luck!

Fortunately, Kloudless now helps you do just that. We are proud to announce that the Calendar Availability endpoint is online and ready to help your application find meeting times that work for any combination of user accounts and their calendars.

Using the Calendar Availability Endpoint

The Calendar Availability endpoint returns all available time windows among a set of calendars that match a specific meeting duration and desired time windows for the meeting. The endpoint also supports requests involving multiple calendars for multiple accounts.

To illustrate, imagine your app has 100 users who have two personal (Google) calendars and two work (Outlook) calendars for a total of four hundred calendars. One call to the Calendar Availability endpoint can retrieve available meeting times (if any) for all 100 users subject to all four of their calendars. Alternatively, you can retrieve available meeting times for just a subset of users and/or their calendars.

Kloudless seamlessly integrates with Google and Outlook Calendar behind the scenes—all you have to provide are Account IDs, Calendar IDs, a meeting duration, and time constraints. Let’s take a look at exactly how to do that.

Formatting the Request and Parsing the Response

Here are the details for the request and response format of the Calendar Availability endpoint based on our docs. You can scroll down further to see concrete examples.

Request Format

URL: https://api.kloudless.com/v1/accounts/{account_id,account_id,…}/cal/availability

Method: POST

Headers:

  • Authorization: Bearer [Token]
  • Content-Type: application/json

Body:

  • calendars: List of Calendar IDs. Uses the default calendar if empty. (Optional)
  • meeting_duration: ISO 8601 format for duration. (Required)
  • constraints: A dictionary of constraint keys and values. (Required)
    • time_windows: List of desired time slots with the following values.
      • start: ISO 8601 datetime format
      • end: ISO 8601 datetime format

Response Format

Headers:

  • Content-Type: application/json

Body:

  • time_windows: List of desired time slots with the following values.
    • start: ISO 8601 datetime format
    • end: ISO 8601 datetime format

The start and end times of each time_window in the response bookend (inclusively) the time periods in which all of the accounts are available given the constraints. Note that the times are returned in the GMT time zone, so you may want to convert to the time zone of your choice.

Concretely, if the response includes the “2 – 5 PM” time window and you wanted a 30-minute meeting, you can safely schedule the meeting to start and end at any time within 2 – 5 PM, inclusive (e.g., 2 – 2:30, 2:01-2:31, …, 4:29-4:59, 4:30-5:00).

Example Usage

Single account, two calendars specified

curl -H 'Authorization: Bearer [TOKEN]' \
    -H 'Content-Type: application/json' \
    -XPOST -d '{
        "calendars": ["ra5werWRsZXQzLcRik5BudGltb6RwwUNnbWFpbC5jb21=",
                      "fa2xvdWRsZXNzLnRlc3QudGltb3RoeUBnbWFpbC5jb20=”],
        "meeting_duration": "PT1H",
        "constraints": {
            "time_windows": [{
                "start": "2017-05-20T08:00:00+07:00",
                "end": "2017-05-20T12:00:00+07:00"
            },{
                "start": "2017-05-21T08:00:00+07:00",
                "end": "2017-05-21T12:00:00+07:00"
            }]
        }
    }' \
    'https://api.kloudless.com/v1/accounts/123/cal/availability'

 

{
  "time_windows": [
    {
      "start": "2017-05-20T02:00:00Z",
      "end": "2017-05-20T04:00:00Z"
    },
    {
      "start": "2017-05-21T03:00:00Z",
      "end": "2017-05-21T04:00:00Z"
    }
  ]
}

In this case, the requestor wants to find an appropriate meeting time for a one hour meeting during the 8 AM – 12 PM GMT+7 time window on either May 20, 2017 or May 21, 2017. The requestor is requiring one account as a participant in the meeting and further specifying two calendars from that account.

Note that the primary calendar within an account is automatically considered if no calendars are provided. But, if any calendars are provided, the primary calendar must be included to be considered. For example, if the primary calendar ID in this case is neither ra5werWRsZXQzLcRik5BudGltb6RwwUNnbWFpbC5jb21= nor fa2xvdWRsZXNzLnRlc3QudGltb3RoeUBnbWFpbC5jb20=, the primary calendar would be ignored when retrieving availability. To also consider the primary calendar, the requestor should provide it as a third calendar in the given calendar list.

The response indicates that any one-hour slot between either 9 AM – 11 AM GMT+7 on May 20, 2017 or 10 AM – 11 AM on May 21, 2017 works for the meeting. Of course, the desired meeting length is exactly as long as the boundaries of the second available time window, so there is only one slot that works on that date.

Multiple accounts, no calendars specified

curl -H 'Authorization: Bearer [TOKEN]' \
    -H 'Content-Type: application/json' \
    -XPOST -d '{
        "meeting_duration": "PT45M",
        "constraints": {
            "time_windows": [{
                "start": "2017-06-15T08:00:00-03:00",
                "end": "2017-06-15T17:00:00-03:00"
            }]
        }
    }' \
    'https://api.kloudless.com/v1/accounts/123,456,789/cal/availability'
{
  "time_windows": [
    {
      "start": "2017-06-15T05:00:00Z",
      "end": "2017-06-15T06:30:00Z"
    },
    {
      "start": "2017-06-15T07:45:00Z",
      "end": "2017-06-15T10:00:00Z"
    },
    {
      "start": "2017-06-15T12:30:00Z", 
      "end": "2017-06-15T14:00:00Z"
    }
  ]
}

Here, the requestor wants to find a time for a 45-minute meeting during the 8 AM – 5 PM GMT-3 time window on June 15, 2017. The requestor is requiring three accounts as participants in the meeting, and since no calendars are specified, is searching for availability based solely on each account’s primary calendar.

Note that you can pass time window start and end times in any time zone (and, in fact, differing time zones within the same request). Furthermore, the calendars to be searched on need not be in the same time zone as the constraints. The important point is that the times are passed as ISO 8601-formatted strings—Kloudless will handle the rest.

The response indicates that any 45-minute slot between 9 AM – 10:30 AM GMT-3, 0:45 AM – 1 PM GMT-3, or 3:30 – 5 PM GMT-3 on June 15, 2017 works for the meeting.

Looking to the Future

The Calendar Availability endpoint provides a simple way to coordinate meetings among multiple calendars and calendar accounts. Going forward, we will monitor usage of the endpoint and listen to your feedback to determine the best way to expand the endpoint’s functionality and flexibility. We already have some great suggestions for making the endpoint more customizable, including:

  • The start_times constraint. If the requestor provides start_times in addition to (or in lieu of) time_windows, the endpoint would return the subset of start_times that work as the start time for the meeting. Essentially, this lets the requestor ask “Which of these exact start times are OK for the meeting?”
  • The ​min_attend_percent constraint. Right now, a time window will only be included in the response if every given account is available in that window—that is, if one relevant calendar in one account is not available during a window, that account is not available and so that window is not returned. With the min_attend_percent constraint, a time window would be included if at least the specified percentage of accounts are available during that window.e
  • The work_or_personal constraint. The requestor could provide a work_or_personal argument to limit the times considered for work or personal hours. This could be especially useful when trying to coordinate across widely varying time zones. For example, a colleague across the world may technically be “available” for a meeting at the time the requestor desires, but that’s of little use if it’s 2 AM for that colleague!

Wrapping Up

We’re excited to rollout the Calendar Availability endpoint to the developers on our platform. What do you like about the endpoint right now, what questions do you have about it, and what do you wish it would do in the future? We would love to hear your thoughts and feedback on Twitter, our developer forums, comments below, or at hello@kloudless.com.

 

Calendar API: Activity Stream

This post is by our engineering intern, Ellen Luo.

Receive real-time updates for Google and Outlook Calendar with the new Events endpoint for the Calendar API!

The Events endpoint allows your application to easily keep track of calendar items that have been added, modified or deleted in your primary calendar. This Events endpoint is part of our broader Events API to monitor activity in connected accounts, and should not be confused with the Calendar Events API that enables access to calendar appointments.

Retrieving activity data with the Events API

To start using the Events API, check the box beside “Collect Events” in the App Details page of the Developer Portal. Kloudless will automatically begin to collect activity data for any newly connected Google or Outlook Calendar accounts. Google Calendar allows real-time updates of changes to users’ accounts so event data is updated in at most 1.5 minutes. For Outlook Calendar, Kloudless queries the service repeatedly to find changes. This means that event data will be updated on average every 15 minutes in our cloud and 1-5 minutes in private installs (configurable).

Requesting activity data

To return a list of all recent events collected, make a request to https://api.kloudless.com/v1/accounts/{account_id}/events/ with your connected account’s ID and Bearer token (or application API Key). A sample request is shown below:

curl -X GET -H 'Accept: application/json' \
    -H 'Authorization: Bearer {token}' \
    'http://api.kloudless.com/v1/accounts/{account_id}/events/'

Parsing the response

The response will contain a list objects of all the events. Each object has a type, which indicates the type of change. Right now, the only events that are detected are “add” (object created), “update” (existing object modified), and “delete” (object deleted). Each event object will have the updated metadata of the object modified. More information about the Events API and sample responses can be found in the Kloudless docs here.

Roadmap

In the future, we will add support for retrieving activity data for calendars other than the account’s primary calendar. We will also be working towards including more specific event types and expanding support to more calendaring APIs. We’re excited to make the Calendar API Activity Stream endpoint available to all developers on our platform and would love to hear any feedback or suggestions on Twitter, the comments below, or at hello@kloudless.com.