Monitoring changes in Salesforce Timothy Liu Published: January 30, 2019 IntroductionKloudless offers a unified Events API for developers to retrieve real-time notifications via Webhooks for any cloud service that we support. We’ve had to build a unique technical implementation for events from Salesforce based on the different options to efficiently track changes. Salesforce recently updated their platform with two new guides on how to retrieve these events in real-time.Salesforce offers the following options:Change Data Capture (Pilot)Platform Events (Pilot)Streaming APIWe’ve provided a brief summary of the different options and desired use cases before diving into CometD the core technology used to retrieve events from Salesforce.Salesforce EventsTo differentiate the following events described with each option, Salesforce uses the following terms:Change Data Capture: A Change EventPlatform Events: A Platform EventStreaming API: A PushTopic or Generic EventWe’ve copied over the feature comparison chart of the different events from Salesforce.Change Data CaptureThe typical use case for Change Data Capture (CDC) is if your organization wants to keep an external data store of events in sync with Salesforce activity. The end result is that your external data store has a perfect copy of all Salesforce event data that you specify. A primary benefit of using CDC is that the field data can also be encrypted with Shield Platform Encryption. Any change in encrypted field values generate change events whereas this is unsupported in the two other options. The event retention period for CDC events is also 3 days.Platform EventsThe major benefit of Platform Events is user customizability. Unlike Change Data Capture and Streaming API, the developer has to build and configure the “Platform Event” that Salesforce will return. In addition, Platform Events has a deeper integration with the Apex platform by being able to publish and subscribe to events via Apex. Therefore, I would recommend using Platform Events if the Apex platform is crucial to your organization’s use case, and you have the developer resources to handle the customization.Streaming APIThe Streaming API is Salesforce’s original mechanism to retrieve events and offers the ability to retrieve both PushTopic and Generic events. A PushTopic event is configured by creating a SOQL query that will return all events that match that configuration. A Generic event is configured to when wanting to send or receive custom notifications that are not tied to Salesforce data changes. Since the primary use cases of CDC and Platform Events revolve around Salesforce data changes, we will mostly be considering PushTopic events. PushTopic events strike a balance between CDC and Platform events since you can customize the type of event data via PushTopics as well as use the default objects without having to create each individual event like in Platform Events.CometD | Bayeaux ProtocolCometD uses a simple publish and subscribe process to receive messages across channels. Salesforce has the following channels to establish a connection:/meta/handshake/meta/connect/meta/disconnect/meta/subscribe/meta/unsubscribeOnce a connection is established, the client can send messages to receive events based on a specific replay ID. Since Salesforce allows all types of events to be returned when using CometD, we’ve provided examples of messages returned for each event type.Generic Event Message { "clientId":"a1ps4wpe52qytvcvbsko09tapc", "data":{ "event":{ "createdDate":"2016-03-29T19:05:28.334Z", "replayId":55 }, "payload":"This is a message." }, "channel":"/u/TestStreaming" }1234567891011{ "clientId":"a1ps4wpe52qytvcvbsko09tapc", "data":{ "event":{ "createdDate":"2016-03-29T19:05:28.334Z", "replayId":55 }, "payload":"This is a message." }, "channel":"/u/TestStreaming"}PushTopic Event Message { "clientId":"2t80j2hcog29sdh9ihjd9643a", "data":{ "event":{ "createdDate":"2016-03-29T16:40:08.208Z", "replayId":13, "type":"created" }, "sobject":{ "Website":null, "Id":"001D000000KnaXjIAJ", "Name":"TicTacToe" } }, "channel":"/topic/TestAccountStreaming" }12345678910111213141516{ "clientId":"2t80j2hcog29sdh9ihjd9643a", "data":{ "event":{ "createdDate":"2016-03-29T16:40:08.208Z", "replayId":13, "type":"created" }, "sobject":{ "Website":null, "Id":"001D000000KnaXjIAJ", "Name":"TicTacToe" } }, "channel":"/topic/TestAccountStreaming"}Platform Event Message { "data": { "schema": "dffQ2QLzDNHqwB8_sHMxdA", "payload": { "CreatedDate": "2017-04-09T18:31:40.517Z", "CreatedById": "005D0000001cSZs", "Printer_Model__c": "XZO-5", "Serial_Number__c": "12345", "Ink_Percentage__c": 0.2 }, "event": { "replayId": 2 } }, "channel": "/event/Low_Ink__e" }12345678910111213141516{ "data": { "schema": "dffQ2QLzDNHqwB8_sHMxdA", "payload": { "CreatedDate": "2017-04-09T18:31:40.517Z", "CreatedById": "005D0000001cSZs", "Printer_Model__c": "XZO-5", "Serial_Number__c": "12345", "Ink_Percentage__c": 0.2 }, "event": { "replayId": 2 } }, "channel": "/event/Low_Ink__e"}Change Data Capture Event Message { "data": { "schema": "IeRuaY6cbI_HsV8Rv1Mc5g", "payload": { "ChangeEventHeader": { "entityName": "Account", "recordIds": [ "<record_ID>" ], "changeType": "CREATE", "changeOrigin": "com.salesforce.core", "transactionKey": "001b7375-0086-250e-e6ca-b99bc3a8b69f", "sequenceNumber": 1, "isTransactionEnd": true, "commitTimestamp": 1501010206653, "commitNumber": 92847272780, "commitUser": "<user_ID>" }, "Name": "Acme", "Description": "Everyone is talking about the cloud. But what does it mean?", "OwnerId": "<owner_ID>", "CreatedDate": "2017-07-25T19:16:44Z", "CreatedById": "<user_ID>", "LastModifiedDate": "2017-07-25T19:16:44Z", "LastModifiedById": "<user_ID>" }, "event": { "replayId": 6 } }, "channel": "/data/ChangeEvents" }1234567891011121314151617181920212223242526272829303132{ "data": { "schema": "IeRuaY6cbI_HsV8Rv1Mc5g", "payload": { "ChangeEventHeader": { "entityName": "Account", "recordIds": [ "<record_ID>" ], "changeType": "CREATE", "changeOrigin": "com.salesforce.core", "transactionKey": "001b7375-0086-250e-e6ca-b99bc3a8b69f", "sequenceNumber": 1, "isTransactionEnd": true, "commitTimestamp": 1501010206653, "commitNumber": 92847272780, "commitUser": "<user_ID>" }, "Name": "Acme", "Description": "Everyone is talking about the cloud. But what does it mean?", "OwnerId": "<owner_ID>", "CreatedDate": "2017-07-25T19:16:44Z", "CreatedById": "<user_ID>", "LastModifiedDate": "2017-07-25T19:16:44Z", "LastModifiedById": "<user_ID>" }, "event": { "replayId": 6 } }, "channel": "/data/ChangeEvents"}Kloudless EventsAlthough Salesforce suggests using the open source EMP Connector, Kloudless decided to implement the Bayeaux Protocol using Python as part of its native stack. Kloudless abstracts away the complexity of connecting to the Streaming API by providing a REST interface to retrieve events. Kloudless combines the best of all three options by being the external data store of events with the customizability. We’ve shown a curl request below that creates a PushTopic, which Kloudless subscribes to for events. curl -XPOST 'Authorization: Bearer [TOKEN]' \ -H 'X-Kloudless-Raw-URI: /services/data/v39.0/sobjects/PushTopic' -H 'X-Kloudless-Raw-Method: POST' \ 'https://api.kloudless.com/v1/accounts/me/raw' \ --data-binary '{"Name": "AccountEvents, "Query": "SELECT name FROM Account", "ApiVersion": 39.0, "NotifyForOperationCreate": True, "NotifyForOperationUpdate": True, "NotifyForOperationUndelete": True, "NotifyForOperationDelete": True, "NotifyForFields": "All"}'12345curl -XPOST 'Authorization: Bearer [TOKEN]' \ -H 'X-Kloudless-Raw-URI: /services/data/v39.0/sobjects/PushTopic' -H 'X-Kloudless-Raw-Method: POST' \ 'https://api.kloudless.com/v1/accounts/me/raw' \ --data-binary '{"Name": "AccountEvents, "Query": "SELECT name FROM Account", "ApiVersion": 39.0, "NotifyForOperationCreate": True, "NotifyForOperationUpdate": True, "NotifyForOperationUndelete": True, "NotifyForOperationDelete": True, "NotifyForFields": "All"}'Retrieve Kloudless events for Salesforce by using our Events API. Here is a sample request below: curl -H 'Authorization: Bearer [TOKEN]' \ 'https://api.kloudless.com/v1/accounts/me/events?cursor=121'12curl -H 'Authorization: Bearer [TOKEN]' \ 'https://api.kloudless.com/v1/accounts/me/events?cursor=121'With Kloudless Enterprise, our customers have sub-minute notifications of activity for a number of cloud services including Salesforce. Kloudless is the ideal solution for your application if you do not want to implement your own pub/sub client, or if you want to have a far simpler integration with Salesforce.