A Look at the Copy API David Thorman Published: May 8, 2014 Disclaimer: This is coming from my personal experience with the Copy API and other cloud storage APIs. It is meant as a summary of the good aspects as well as the “gotchas” that I have encountered. Hopefully it will provide some insight into decisions that were made when designing the Kloudless API. This blog post does not reflect the opinions of my employer.About Copy the ProductCopy is a relative newcomer to the cloud storage ecosystem and is backed by Barracuda Networks which is a rather large company that specializes in security and storage appliances.Copy offers a solid starting plan with 15GB of storage for free and touts that it works on just about every platform. They encrypt all of your data in transit and at rest with AES 256 encryption, which is pretty standard. One unique offering is their Fair Storage option where one can divide up some amount of storage between you and your friends and each pays proportionally for what they use. That is all product side stuff and it looks good, but I care more about the API.The Copy APICopy’s API is simple and relatively easy to work with because it uses standard technologies and design practices:Oauth1.0a for authenticating and authorizing requestsJSON for its data serializationRESTful structure of resourcesstandard HTTP actions for resource interactionsIt also has the following features which one would expect from a cloud storage service:CRUD for file and folder resourcesFile uploads support automatic rename and overwriteCreation of links for sharingPermissions on linksGeneration of thumbnails for filesAn ‘Inbox’ that is used to interact with files that are shared with youPrevious versions of files (coming soon)SDKs in the following languages:C++PHPNode.js.NET (third-party)OSX/iOS (Objective-C, third-party)Go Lang (third-party)From a high level it looks pretty good and has the features you would need if you were building an application on top of it. There are some design choices that are worth noting when comparing it to other cloud storage services:Resources are identified by their path, not a static, globally unique identifier like Box or SharefileFile uploads can be done either by a form or by posting the file contents to the path that you wish the file to have, in each case you have to specify a path in the urlFolders are created by POSTing to the url with the path that you wish the new folder to haveAPI versioning is done based on a header, rather than in the url (not too big of a deal)Several issues can arise from using file paths as the identifier for files in the url. One of the more obvious ones is that if you have stored that id and you move/rename the resource, then your reference becomes invalid. This issue can also be found in Dropbox’s API, so Copy ins’t the only one with this problem. A less obvious issue with having paths in the url is that you have to make sure that they are encoded properly, since folder and file names can contain special characters like test special chars ~!@#$%^&*()_-+={}|]["';?.,.txt and unicode like Üñįčødë Téßt2.txt. Programming languages such as Python 2.7 aren’t that great at handling unicode strings to begin with and making sure that all the encodings are correct can be something of a hassle, but I digress. This is compounded by the fact that the restrictions that Copy places on the folder and file names is not documented. Below is some python code that I have written to deal with this: import urllib import re def sanitize_name(self, name): """ This sanitizes a file/folder name into something safe to append to an existing folder path to make a new resource. It assumes that the input name is a unicode string. """ if name in ['.', '..']: name = re.sub(r'[.]', '-', name) name = re.sub(r'[:\\]', '', name) name = re.sub(r'[/]', ':', name) return urllib.quote(name.encode('utf8'), safe='~_-.*()!') 123456789101112131415import urllibimport redef sanitize_name(self, name): """ This sanitizes a file/folder name into something safe to append to an existing folder path to make a new resource. It assumes that the input name is a unicode string. """ if name in ['.', '..']: name = re.sub(r'[.]', '-', name) name = re.sub(r'[:\\]', '', name) name = re.sub(r'[/]', ':', name) return urllib.quote(name.encode('utf8'), safe='~_-.*()!') The form-based upload in the Copy documentation is somewhat new, but I have run into unicode file name encoding issues with services like Box that pull the file name from the MIME field header. When using a version of python-requests greater than 1.2.3, it (the included version of urllib3) encodes the name of the file according to RFC 2231. This caused uploads to fail because Box’s servers didn’t handle it. We eventually ended up using an undocumented feature of the Box API to specify the metadata in a JSON dictionary (this is why the Kloudless API uses format it does for file uploads).Choosing a specification for authenticing users authorizing requests depends largely on security and convenience trade offs. Copy uses 3-legged OAuth1.0a. They document the procedure well and follow the standard which is great. The trade off is that requests with OAuth1.0a can be a pain when you don’t have a library, mostly because it involves a cryptographic signature. This is in contrast to OAuth2.0 which simply gives the application a bearer token to authorize the requests (this is aruguably less secure, but it assumes the transport layer is secure). In most cases, there is no downside to using OAuth1.0a as long as you are using a mainstream language, but it makes a little more tricky to test out the requests using cURL.The Inbox is quite limited from the Copy API’s perspective because one cannot directly access the contents of files in it in the same way as the normal Copy file system. This has seriously limited how useful it is to our GMail extension and our platform. There are plans to expose these interactions, but currently it is not possible without first syncing them to the main filesystem.One last thing that might seem somewhat funny is the absence of file/folder copying method. The current work around if you want to make a copy of a file through the API, one must download the file and re-upload it. There will be an endpoint exposed in the future, but it isn’t too bad as long as you aren’t trying to copy large numbers of files.How the Kloudless Platform Can HelpThere are eccentricities to every API and Copy is no exception. The Kloudless API is meant to allow the developer to provide flexible cloud storage functionality to their users in their applications without having to deal with these details. We provide a uniform interface to nine different cloud storage services through a single, simple RESTful API. Check it out and let us know what you think!