1. About

Streamdata.io is a real-time data distribution platform that allows you to either poll a Rest API or push messages directly from your backend. Streamdata.io enables you to delegate the complexity of real-time data push while dramatically reducing bandwidth consumption through incremental data streaming. Go beyond the traditional request/response mechanism and the limits that come with this approach. Streamdata.io uses Server-Sent-Events in order to provide its push feature. Instead of requesting data every time, the client subscribes to receive real-time updates coming from the server in an automated fashion.

2. Features

  • Push: Updates are pushed to the client using Server-Sent Events (SSE). By providing fallback mechanisms, Streamdata.io can also work with older browsers.

  • Incremental updates: Replacing polling by push is not enough to minimize data streams. That’s why Streamdata.io only pushes the part of data that has changed thanks to JSON Patch.

  • Cache: To reduce server load and latency, Streamdata.io relies on an In-memory Data Grid to efficiently cache data across its cluster.

3. Docker Deployment

The Streamdata.io platform can be deployed as a multi-container Docker application. Each microservice of Streamdata.io platform runs inside a docker container. The deployment of all microservices is automated thanks to docker compose tool.

The Streamdata.io Docker platform is delivered as a Beta version. This version might not provide the same level of performance or compatibility as a generally available (GA) version. Features included might change frequently and might be discontinued with short notice. It is limited to a maximum of 10 concurrent client connections and is not intended for use in a production environment.

3.1. Pre-requisites

3.1.1. Software requirements

  • Linux: Docker version 18.06.0-ce+ / docker-compose version 1.21.2+

  • macOS: Docker version 18.06.0-ce+ / docker-compose version 1.22.0+

  • Microsoft Windows 10

    • Windows toolbox: Docker version 18.03.0-ce+ / docker-compose version 1.20.1+

    • Windows 10 pro: Docker version 18.06.0-ce+ / docker-compose version 1.21.2+

    • Virtualization must be enabled in the BIOS. Everything you need to know about this topic is explained in docker documentation.

3.1.2. Docker minimal resources

  • RAM: 4GB

  • CPU: 2

  • How to set the hardware requirements on your docker installation

Once the RAM is configured, the 4GB must be free and available on your machine.
The default configuration will use the following ports on your localhost: 8000, 8001 and 6080. On windows toolbox, the ports are allocated on the virtual machine itself.

3.2. Installation procedure

Follow the steps below in order to get your Streamdata.io platform running:

git clone https://github.com/streamdataio/streamdataio-docker
  • Using command line, go to the directory where the docker-compose.yml is located

  • Get you activation key here

  • Copy and paste your activation key in the .env file

  • Run the following command: docker-compose up -d

  • Expected output:

$ docker-compose up -d
Creating network "streamdataio-docker_default" with driver "bridge"
Creating facade                   ... done
Creating database                 ... done
Creating http-poller-publisher    ... done
Creating proxy-sse                ... done
Creating router                   ... done
Creating http-post-publisher      ... done
Creating kong                     ... done
Creating apps                     ... done
Creating topics                   ... done
Creating kong-migration           ... done
Creating kong-configuration       ... done
Creating auth                     ... done
Creating users                    ... done
  • Now that all the containers are created, the process of starting each component can take a couple of minutes. The platform will be up and running when all the containers are healthy. The health of a container can be checked thanks to the docker ps command (or docker-compose ps command).

Wait for facade and users services to be healthy:

CONTAINER ID        IMAGE                                            COMMAND                  CREATED             STATUS                       PORTS                                             NAMES
11cbab388339        streamdataio/streamdataio-users                 "/bin/bash /home/str…"   17 minutes ago      Up 16 minutes (healthy)                                                     users
014f530c0255        streamdataio/streamdataio-auth                  "/bin/bash /home/str…"   17 minutes ago      Up 17 minutes (healthy)                                                     auth
04dccce495b4        kong:0.13.0-alpine                              "/docker-entrypoint.…"   17 minutes ago      Up 13 minutes (healthy)   0.0.0.0:8000-8001->8000-8001/tcp, 8443-8444/tcp   kong
3ef41751e39f        streamdataio/streamdataio-topics                "/bin/bash /home/str…"   17 minutes ago      Up 17 minutes (healthy)                                                     topics
6e73b36aab34        streamdataio/streamdataio-apps                  "/bin/bash /home/str…"   17 minutes ago      Up 17 minutes (healthy)                                                     apps
4368eeae68a4        streamdataio/streamdataio-router                "/home/streamdataio/…"   17 minutes ago      Up 17 minutes (healthy)                                                     router
0c6ab83f556b        streamdataio/streamdataio-http-poller-publisher "/home/streamdataio/…"   17 minutes ago      Up 17 minutes (healthy)                                                     http-poller-publisher
5ed70c7b7372        streamdataio/streamdataio-facade                "/home/streamdataio/…"   17 minutes ago      Up 17 minutes (healthy)                                                     facade
bf1d4aab525c        streamdataio/streamdataio-http-post-publisher   "/home/streamdataio/…"   17 minutes ago      Up 17 minutes (healthy)                                                     http-post-publisher
e68be2e89ebb        streamdataio/streamdataio-proxy-sse             "/home/streamdataio/…"   17 minutes ago      Up 17 minutes (healthy)   0.0.0.0:6080->6080/tcp                            proxy-sse
ee8beadcf79e        cassandra:3                                     "docker-entrypoint.s…"   17 minutes ago      Up 17 minutes (healthy)   7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp       database
  • Test that the platform is up and running:

You should now be able to use Streamdata.io platform and make API calls. Please refer to the Management API for the complete list of available APIs.

Once the Streamdata.io Platform is successfully deployed, the OpenAPI specification of the Management API is available here: http://localhost:8000/api/v1/api-browser/index.html

For instance, you can try to create a new user:

curl -v -X POST "http://localhost:8000/api/v1/users" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"email\":\"john.doe@streamdata.io\",\"password\":\"password123\",\"first_name\":\"John\",\"last_name\":\"Doe\",\"language\":\"fr_FR\"}"

Response should be a HTTP 201 created.

If you are using docker toolbox on windows, you must use the IP of your virtual machine instead of localhost. This IP is displayed in the docker toolbox terminal just before docker has finished starting.

3.3. Update to latest version

When a new version of the containers is available on our dockerhub, you can run docker-compose pull to update your local docker images.

3.4. Docker Compose Commands

The table below summarizes the most useful commands of docker-compose. For a complete list, please refer to the command-line reference page.

Command Description

docker-compose up -d

Run containers in detached mode (background).

docker-compose down

Stops and removes containers and networks created by up.

docker-compose start

Starts existing containers for a service.

docker-compose stop

Stops running containers without removing them. They can be started again with docker-compose start.

docker-compose restart

Restarts all stopped and running services.

docker-compose pull

Pulls an image associated with a service defined in a docker-compose.yml.

3.5. Admin user

An admin user is automatically created when the platform is initialized.

email Password

docker-admin@streamdata.io

streamdata

The OpenAPI spec of the Admin API is available here: http://localhost:8000/api/v1/api-browser-admin/index.html

3.6. Exposed Endpoints

The table below summarizes the configured endpoints on which the Streamdata.io platform listens by default.

Description Endpoint

SSE Proxy - Auto-provisioning

http://localhost:6080/{{URL}}

SSE Proxy - Manual Provisioning

http://localhost:6080/v1/topics/{{topic-id}}

Management API

http://localhost:8000/api/v1

Management API OpenAPI Spec

http://localhost:8000/api/v1/api-browser/index.html

Admin API OpenAPI Spec

http://localhost:8000/api/v1/api-browser-admin/index.html

HTTP Post Publisher API

http://localhost:8000/publish/v1

HTTP Post OpenAPI Spec

http://localhost:8000/publish/v1/api-browser/index.html

3.7. Additional configuration

Feel free to open and edit the docker-compose.yml to change the platform configuration.

3.7.1. Change port mapping

If you have a port conflict on your localhost, you can change the port mapping.

3.7.2. Change environment variable

It is possible to override microservice configuration by passing environment variables in the docker compose file:

Below, variables you can change:

Variable Name Component Description

SDIO_admin_email

users

email for admin user

   users:
    ...
    environment:
      - SDIO_admin_email=myadmin@mycompany.com
Environment variables are case sensitive.

4. Concepts

4.1. The Basics of a Pub/Sub Service

A publish/subscribe (Pub/Sub) service is a messaging service where the senders of messages are decoupled from the receivers of messages.

There are several key concepts in a Pub/Sub service:

  • Message: the data that transits through the service.

  • Topic: an entity that represents a feed of messages.

  • Publisher: creates messages and send them in a topic.

  • Subscriber: subscribes to a topic to receive published messages.

architecture

4.2. How it works

  1. A publisher application creates a topic and sends messages to it. Each message must contain a JSON payload that describes its content.

  2. Messages are stored in an In-Memory Data Grid until they reached their TTL.

  3. Messages are then forwarded to all the subscribers of the topic. Each subscriber receives the messages through its Server-Sent Events connection.

5. Publishing

5.1. Publisher Type

Streamdata.io supports different publisher types. Each topic created on the Streamdata.io platform must be associated with a type of publisher. When creating your topic, you must set the type attribute in the topic's configuration.

{
  ...
  "publisher": {
    "type": "http-poller|http-post",
    ...
  }
  ...
}

The table below lists the different types of publishers available:

Publisher Type Id Description

HTTP Poller Publisher

http-poller

A publisher that polls a target API and publishes its response in the associated topic.

HTTP Post Publisher

http-post

A publisher that accepts HTTP Post requests from a third-party component and publishes received content in the associated topic.

5.2. Payload Type

Regardless of its type, each publisher may have different capabilities when it comes to sending data to Streamdata.io. Streamdata.io supports different types of payload. This configuration must be defined through the attribute type under payload in the topic's configuration.

{
  "name": "myTopic",
  ...
  "publisher": {
    ...
    "payload": {
      "type": "item|snapshot"
    }
   }
}

If you don’t set payload when creating a topic Streamdata.io will set the following by default:

{
  ...
  "publisher": {
    ...
    "payload": {
      "type": "snapshot"
    }
   }
}

5.2.1. Item Publisher

A Publisher that publishes parts of a dataset over time.

This publisher must send the data as an Array containing one or more items. Each item must be represented as a JSON Object containing a list of key/value pairs.

Examples

[{
 "Symbol":"AUDUSD",
 "Bid":0.74709,
 "Mid":0.74713,
 "Ask":0.74717
}]

To select this mode, you must set the type attribute to item. You must also define the attribute of the item that uniquely identifies it so that the Streamdata.io platform can distinguish new items from already published items. This configuration must be defined through the attribute item_id_path under payload node in the topic's configuration.

{
  "name": "myTopic",
  ...
  "publisher": {
    ...
    "payload": {
      "type": "item",
      "item_id_path": ".Symbol"
    }
  }
}
  • The configuration is defined upon topic creation and cannot be changed afterwards.

  • The value of item_id_path is case sensitive.

  • The first character of the item_id_path expression defines the separator that will be used in the rest of the expression. See Nested Identifier section for more details.

In the example above, the topic should be configured with an item_id_path=.Symbol.

All items published in a topic MUST contain an unique identifier. Items published without any identifier will be discarded. Based on the configuration above, the following item would be discarded as Symbol attribute is missing.
{
 "Bid":0.74709,
 "Mid":0.74713,
 "Ask":0.74717
}
Nested Identifier

If the attribute identifying the item is located under a nested node, you can define its path in the item_id_path attribute.

Given the following items:

{
 "Bid":0.74709,
 "Mid":0.74713,
 "Ask":0.74717,
 "Identifier": {
    "Symbol": "AUDUSD"
 }
}

The proper value for the item_id_path attribute would be .Identifier.Symbol

To avoid conflicts between special characters that may be present in your own content and the separator used in the path expression, you can define the seperator to use in the first character of the expression.

The following expression is also valid

"item_id_path": "|Identifier|Symbol"
Item Update mode

The Item Update mode defines how Streamdata.io will react when a field is no longer present in a published item.

Currently Streamdata.io only handles the PATCH mode.
PATCH mode

The publisher publishes several publications related to the same item. The topic in which content is published is configured to use the Symbol attribute as key identifier.

Publication #1 (t0)

{
 "Symbol":"AUDUSD",
 "Bid":0.74709,
 "Mid":0.74713,
 "Ask":0.74717
}

Publication #2 (t0+1)

{
 "Symbol":"AUDUSD",
 "Ask":0.74720
}

Streamdata.io will cache a patched version of the item. Meaning that the fields not present in the last update are removed, new fields are added and existing fields are updated.

Resulting Cached Item

{
 "Symbol":"AUDUSD",
 "Ask":0.74720
}
MERGE mode
Not Supported yet.

The publisher publishes several publications related to the same item as described below. The topic is configured to use the Symbol attribute as key identifier.

Publication #1 (t0)

{
 "Symbol":"AUDUSD",
 "Bid":0.74709,
 "Mid":0.74713
}

Publication #2 (t0+1)

{
 "Symbol":"AUDUSD",
 "Ask":0.74717
}

Streamdata.io will cache a merged version of the item. Meaning that the absence of a field is not consider as a removal. New fields are added and the last published value of each field is kept in Streamdata.io cache.

Resulting Cached Item

{
 "Symbol":"AUDUSD",
 "Bid":0.74709,
 "Mid":0.74713,
 "Ask":0.74717
}

5.2.2. Snapshot Publisher

A Publisher that publishes an updated version of a full dataset over time.

You can select this mode by setting the type attribute to snapshot in your topic's configuration.

{
  "name": "myTopic",
  ...
  "publisher": {
    ...
    "payload": {
      "type": "snapshot"
    }
   }
}

Whenever a new version of a JSON document (a.k.a snapshot) is published, a SSE Event containing the JSON patch operations to be applied is emitted to all subscribers of the topic.

Let’s take a real life example, a publisher publishes the latest top news (for the sake simplicity let’s say our source publishes only the two last news).

Publication #1 (t0): first snapshot. It will be sent as is to subscribers.

[{
   "id": "acb07740-6b39-4e8b-a81a-0b678516088c",
   "title": "94% of Banking Firms Can’t Deliver on ‘Personalization Promise’",
   "date": "2018-09-10-T10:13:32",
   "abstract": "One of the strongest differentiators ..."
},{
   "id": "0c5b5894-a211-47de-87a8-c7fa3ce3dfa2",
   "title": "Would you trust your salary to start-up",
   "date": "2018-09-10-T09:59:32",
   "abstract": "We take a closer look at how safe..."
}]

Publication #2 (t0+1): the payload is exactly the same as the previous publication, nothing is sent to subscribers.

Publication #3 (t0+2): the oldest headline is gone and a new headline is added on top.

[{
   "id": "55d21525-af74-4b9f-944a-e43a48147d80",
   "title": "How to Navigate Competing Regulations: Unintended Consequences of GDPR",
   "date": "2018-09-10-T10:14:18",
   "abstract": "GDPR and MiFID II can create a tension between retention ..."
},{
   "id": "acb07740-6b39-4e8b-a81a-0b678516088c",
   "title": "94% of Banking Firms Can’t Deliver on ‘Personalization Promise’",
   "date": "2018-09-10-T10:13:32",
   "abstract": "One of the strongest differentiators ..."
}]

As a result subscribers to the topic will receive a JSON patch like so:

[{
   "op":"add",
   "path":"/0",
   "value": {
      "id":"55d21525-af74-4b9f-944a-e43a48147d80",
      "title":"How to Navigate Competing Regulations: Unintended Consequences of GDPR",
      "date":"2018-09-10-T10:14:18",
      "abstract":"GDPR and MiFID II can create a tension between retention ..."
   }
},{
   "op":"remove",
   "path":"/2"
}]
This is the default behavior, be aware that JSON patch is not the only option. See Subscription delivery modes for more details.

6. Subscribing

To receive messages published to a topic, your subscriber application must initiate a connection with Streamdata.io server. Once established, this persistent connection will enable the Streamdata.io Server to push data to the subscriber application as soon as it is published in the topic.

6.1. Transport Protocol

Streamdata.io relies on Server-sent Events (SSE) technology for real-time data Push.

Read our blog post SSE vs. WebSockets to have a better understanding of why we chose SSE rather than Websocket.

6.2. Response format

Streamdata.io response is JSON (and JSON Patch) over SSE. Don’t worry it’s not rocket science, after reading this chapter, you will know just enough to understand the result of a cURL command targeting Streamdata.io.

6.2.1. SSE, the way Streamdata.io uses it

When you connect to an SSE server you will received a HTTP 200 OK code like a regular successful connection to an HTTP server. But the connection will remain alive and everything will happen afterwards, including errors (e.g. authentication errors, bad requests). As long as the client or server does not end the connection, it will remain alive.

SSE is text-based protocol. This is what you get when the server responds to a connection (metadata, headers, status codes are omitted on purpose):

id: 00ae73f5-5349-40c4-91b6-2e58a36b5365                                         (1)
event: snapshot                                                                  (2)
data : [{                                                                        (3)
   "id": "acb07740-6b39-4e8b-a81a-0b678516088c",
   "title": "94% of Banking Firms Can’t Deliver on ‘Personalization Promise’",
   "date": "2018-09-10-T10:13:32",
   "abstract": "One of the strongest differentiators ..."
},{
   "id": "0c5b5894-a211-47de-87a8-c7fa3ce3dfa2",
   "title": "Would you trust your salary to start-up",
   "date": "2018-09-10-T09:59:32",
   "abstract": "We take a closer look at how safe..."
}]
1 id identifies the message.
2 event is the name of the event. Streamdata.io uses three types of events : snapshot, patch and error.
3 data is the body of the message, always in JSON, in that case a JSON array.

Those three field are always there and represent a single message called "event".

  • The snapshot event is only emitted once just after being connected (unless you explicitly ask for snapshots only). Refer to Subscription delivery modes for details.

  • An error event ends the connection in most cases. Refer to Errors for details.

6.2.2. JSON Patch

After the snapshot has been emitted, it will be followed by patch events as soon as Streamdata.io detects a change in the published content:

id: 37740aa3-3629-41c4-9a7f-24a1347383eb
event: patch                                (1)
data: [{"op":"remove","path":"/1"}]
1 Example of a patch sent by Streamdata.io

What is a patch? It is an JSON document that gives you the difference between two JSON documents. It is represented by an array of operations to apply to the previous version of the document.

An operation is made of two to three fields :

  • op : the operation type

  • path where the operation applied in the document (JSON Pointer)

  • value (optional) value to apply: a raw JSON literal, object or array

operation type has a value

add

yes

replace

yes

remove

no

In the patch example above, there is only one operation, the patch indicates that the second row has to be removed. A patch can be applied to an existing document to alter it (numerous of libraries can take care of that). JSON patch enables Streamdata.io to save bandwidth by pushing only the differences between two version of published content.

Streamdata.io provides Client SDKs and Samples in many different languages to help you handle connection, SSE payload decoding and patch processing.

6.3. Subscription delivery modes

The subscriber can choose between different modes of delivery that determine how the data will be transmitted to him. The delivery mode is defined by the subscriber upon his subscription request. Thus, each subscriber of the same topic can decide which mode is best suited to his needs. All subscription delivery modes rely on an underlying Server-Sent Events connection.

6.3.1. Snapshot + JSON Patch

In this mode, Streamdata.io will send upon the initial connection a full snapshot of all subscribed items. Subsequent events sent to the subscriber will only contain the changed fields in the form of an array of JSON Patch operations.

For example, in the context of a brokerage app, if a user subscribes to 10 different symbols, each symbol contains different fields such as identifier, last, bid, ask. But only a few of them really change at every market tick. Streamdata.io automatically computes the delta and sends the corresponding JSON patch operations to the subscriber avoiding to resend fields that have not changed.

To enable this mode, the subscriber must add an Accept: application/json-patch+json header in its subscription request.

This is the default mode which will be used if no Accept Header is found in the subscription request. This mode can also be enabled with headers Accept: text/event-stream or Accept: */*

6.3.2. Snapshot Only

In this mode, Streamdata.io will send you the entire updated content, each time a message is published in the topic. To use this mode, add Accept: application/json header.

Use this mode for content which is infrequently and fully updated.
Using application/json supersedes application/json-patch+json. Thus subscribing with header Accept: application/json-patch+json, application/json will enable Snapshot Only mode.

6.4. Item filtering

Upon subscription request, the subscriber can subscribe to specific items published in the topic. This subscription-based filtering makes Streamdata.io able to support different routing model.

6.4.1. Unicasting

Published messages target specific subscribers.

unicast

6.4.2. Multicasting

Published messages target a group of subscribers.

multicast

6.4.3. Broadcasting

Published messages target all subscribers.

broadcast

6.4.4. Item Filtering configuration

When a topic is configured to accept item publishing (see section Item Publishing), the subscriber can provide the list of items to which he wishes to subscribe via a request parameter. The name of the query parameter used by Streamdata.io for filtering items is X-Sd-Filter. The subscribers must then provide a list of identifiers as value of this query param. Any updates of this attribute will be taken into account for new subcribers only.

For example, the subscriber will be able to subscribe to the topic with request:

curl "http://localhost:6080/v1/topic/{{topicId}}?X-Sd-Filter=A,B"

6.5. Connection Heartbeat

Legacy proxy servers are known to, in certain cases, drop HTTP connections after a short timeout. To protect against such proxy servers, Streamdata.io sends to the client a comment line (starting with a ':' character) every 10 seconds.

When no change is detected by Streamdata.io, you will get those heartbeats repeatedly until a event is sent.

6.6. Compression

Server-Sent Events flow can be compressed on demand using gzip and/or deflate methods. An example of how to use the X-Sd-Compress header:

curl "https://proxy.streamdata.io/http://stockmarket.streamdata.io/v2/prices" -H "Accept-Encoding: gzip, deflate" -H "X-Sd-Compress: true" --compress
If this header is not provided, the default behavior is NOT to compress the data flow.

6.7. Security

6.7.1. TLS Encryption

Streamdata.io uses Transport Layer Security (TLS) to encrypt messages while transmitting them through the Internet. Using TLS ensures that client messages are protected when being sent to and from Streamdata.io. This prevents intercepted messages from being viewed by unauthorized parties.

6.7.2. Subscriber Key

Streamdata.io provides every user’s application with a unique key to authenticate subscription request to a topic. All topics in the same app share the same subscription key. The Subscriber Key enables Streamdata.io to tie a request to a specific account in order to monitor traffic, enforce quotas, handle billing and manage access control. Thus, you must include a Subscriber Key with all API requests that are passed through Streamdata.io.

If you believe that your key has been compromised, e.g. if you notice suspicious activity in your console traffic reports, then Streamdata.io recommends renewing it.

The Subscriber Key replaces the former App token that becomes deprecated.
Acquiring and Changing Subscriber Key

If you need to renew your Subscriber Key:

Simply perform a PUT request on the /apps/{{app_id}}/subscriber_key.

Using Subscriber Key

In order to provide your Subscriber Key when connecting to Streamdata.io, you can either pass it through a query param or a request header.

Example:

via query param
curl "http://localhost:6080/http://stockmarket.streamdata.io/v2/prices?X-Sd-Subscriber-Key={{SUBSCRIBER_KEY}}"
via header
curl "http://localhost:6080/http://stockmarket.streamdata.io/v2/prices" -H X-Sd-Subscriber-Key: {{SUBSCRIBER_KEY}}"
The Subscriber Key is only required for production environment.

6.8. Errors

When a problem occurs during subscription, an error message is sent. Example:

{
  "datetime":"2018-09-03T13:16:02.120Z","code":40110,"category":"ClientException","message":"Credentials error: your subscriber key is invalid","status":2007,"sessionId":"e23f2fff1f3672f23b033b48566ea1b1"
}
It contains multiple information:
  1. code, category and message can be used to warn the end user (fine management of errors)

  2. datetime, sessionId are useful for a support request

  3. status is only for retrocompatibility with v1 (deprecated)

For an automatic error processing, only code and category data must be used.

There are 3 categories of error: ClientException for client errors, ServerException for internal errors in Streamdata.io platform and TopicException for all errors related to the publication in a topic.

6.8.1. Client Errors

This class of error is intended for situations in which the error have been caused by the client (Subscriber).

{
  ...
  "code":40XXX,"category":"ClientException",
  ...
}
Error Code Description

40000

Bad request

40001

Invalid credentials format

40100

Unauthorized

40110

Credentials error

40111

Credentials revoked

40112

Credentials expired

40120

Account disabled

40121

Account restricted (connection limits exceeded)

40122

Account blocked (message limits exceeded)

40400

Resource not found

40401

Topic not found

40500

Method not allowed

6.8.2. Server Errors

This class of error is intended for situations in which the server is aware that it has encountered an error or is otherwise incapable of performing the request.

{
  ...
  "code":50XXX,"category":"ServerException",
  ...
}
Error Code Description

50000

Internal error

50001

Timeout error

6.8.3. Topic Errors

This class of error is intended for situations in which the error seems to have been caused by the Publisher (Data source).

{
  ...
  "code":60XXX,"category":"TopicException",
  ...
}
Error Code Description

60000

Topic in error

60001

Data source in error

60100

Data source connection error

60101

Data source credentials error

60102

Data source credentials revoked

60103

Data source credentials expired

60400

Data source host unknown

61500

Unsupported data source format

7. Client Resources

Once you have successfully installed your Streamdata.io Platform, you will be able to easily add real-time data in your web or mobile apps. You can access all resources needed to help you connect your client application to the Streamdata.io Platform from our Github.

As Streamdata.io relies on Web Standards, we do not require the client application connecting to the platform to use any proprietary SDK. You are free to use the Server-Sent Event and JSON Patch libraries of your choice available in your preferred language.

You will find below, some examples of client applications implemented in different languages.

7.1. JavaScript

You will find sample applications in different Javascript frameworks such as Angular, Vue.js or Node.js in our JS GitHub repo

We provide an Javascript SDK that includes:

  • A polyfill for older browsers that do not support SSE. For more details, refer to Can I use.

  • An optional Hash-based Message Authentication (HMAC) mechanism to enhance your requests security.

  • A workaround solution to pass extra request Headers as it is not supported in the SSE specification.

For more information, see our Javascript SDK documentation.

7.2. Java

Check out our Java sample app for a server to server integration.

7.3. Android

You can find Android sample apps on our GitHub repo

7.4. iOS

We have also published an iOS sample that shows feeds from the New York Times API through Streamdata.io here

8. HTTP Poller Publisher

Streamdata.io provides a proxy-as-a-service that instantly turns any request/response API into a real-time event-driven data feed.

Polling describes the mechanism used to retrieve data from an API: the client first needs to send a request to a server and the server responds by sending the requested data. Since it is not possible for the client to know when the API data is updated, it usually sends requests as often as possible to try to stick to reality and ends up using a lot of bandwidth and resources to receive the same data several times.

Go beyond the traditional request/response mechanism and the limits that come with this approach. Streamdata.io uses Server-Sent-Events (SSE) in order to provide its push feature. Instead of requesting data every time, the client subscribes to receive updates coming from the server in an automated fashion. That’s streaming. No more polling.

http poller connector overview

8.1. Requirements

8.2. How to create a Streamdata.io App

In order to create an app, you must perform a POST request on the /apps endpoint. When creating an app, you can define a default configuration that will be applied to all the topics linked to a HTTP Poller created under the app. The following example shows a request body that provides a default configuration:

{
  "name": "MyApp",
  "auto_provisioning_allowed": true,
  "client_request_signature_enabled": false,
  "default_configurations": {
    "http-poller": {
      "polling_frequency": 5000
    }
  }
}

8.3. Provisioning modes

When creating an App, you can define whether or not you accept that topics can be created on the fly upon user subscription.

8.3.1. Auto-provisioning

By setting the auto_provisioning_allowed attribute to true, you allow any user in possession of the App’s Subscription Key to create a topic on the fly by performing a subscription request. This option can be useful when you do not know in advance the target API that will be used by the subscribers.

For example, the following subscription request will automatically create a topic linked to a preconfigured HTTP Poller.

  • The target URL will point to http://stockmarket.streamdata.io/v2/prices.

  • The frequency at which Streamdata.io will query the target API will be the one defined by the polling_frequency attribute in the app’s default configuration.

  #$ curl -v "http://localhost:6080/http://stockmarket.streamdata.io/v2/prices"
Auto-provisioned topic are created when the first client subscribes and are automatically deleted when the last client unsubscribes.

8.3.2. Manual provisioning

You can also create a topic through Streamdata.io Managment API.

To do so, perform a POST request on /apps/{app_id}/topics endpoint.

The following example shows a request body that creates a Topic with a HTTP Poller configured with a target API pointing to http://stockmarket.streamdata.io/v2/prices and a polling frequency of 5s:

{
  "name": "MyHttpPollerTopic",
  "publisher": {
    "type": "http-poller",
    "config": {
      "url": "http://stockmarket.streamdata.io/v2/prices",
      "polling_frequency": 5000
    }
  }
}
Configuration entries
Configuration entry Mandatory Default value Description

url

yes

n/a

Target URL that Streamdata.io will request

polling_frequency

no

provided by App default config or 2000ms

Frequency at which the target URL will be requested (in milliseconds)

8.4. Target API Authentication

Streamdata.io supports target API using the different Authentication types:

  • Query Parameters

  • HTTP Headers

  • HTTP Basic

  • OAuth 2.0

When any of the following is provided by the customer in its subscription request, Streamdata.io forwards it to the target API.

8.4.1. Query Parameters

The Query Parameters method lets you add several authentication parameters to your URL.

https://api.yourdomain.com/v1/{endpoint}?QueryParameter=Authentication

8.4.2. HTTP Headers

The HTTP Headers method lets you add several header parameters to your API.

Auth-Header: 'Some Value Parsed by the Server'

8.4.3. HTTP Basic

The HTTP Basic method follows the HTTP Basic Authentication protocol of Username:Password.

Authorization: Basic EWxhUGRpbjpvbGVuIHNlh3FtZQ

This method will automatically require Username and Password in the Authentication section of the test console.

9. HTTP Post Publisher

Using a Streamdata.io HTTP Post Publisher allows any third-party component to publish data to a Streamdata.io Topic through the HTTP POST endpoint exposed by Streamdata.io platform. You can use this type of publisher to publish data from your backend, from your client app or integrate Streamdata.io platform with an external service that provides a Webhook API.

A WebHook is an HTTP callback: an HTTP Post that occurs when something happens; a simple web event-notification.
http post connector overview

9.1. Requirements

9.2. How to create a Streamdata.io App

In order to create an app, you must perform a POST request on the /apps endpoint.

{
  "name": "myApp",
  "default_configurations": {
    "http-post": {
      "force_publishing": true
    }
  }
}

9.3. How to create a Streamdata.io Topic linked to a HTTP Post Publisher

In order to create a Streamdata.io Topic linked to a HTTP Post Publisher, you must perform a POST request on /apps/{app_id}/topics endpoint.

Example of creating a Streamdata.io Topic:

{
  "name": "myTopic",
  "publisher": {
    "type": "http-post",
    "config": {
      "force_publishing": true
    }
  }
}
Configuration entries
Configuration entry Mandatory Default value Description

force_publishing

no

true

Allow to force publishing data into the related topic even if there is no subscribers connected.

As force_publishing is optional but parent attribute config is mandatory, config value can be an empty JSON object {}.

9.4. How to publish to a Streamdata.io Topic

In order to publish to your Streamdata.io Topic, you simply have to perform a POST request against the Streamdata.io HTTP Post API. Refer to the OpenAPI specification for more details.

Below an example using cURL command:

curl -v http://localhost:8000/publish/v1/topics/{{SDIO_TOPIC_ID}} \
-H "Content-Type: application/json" \
-H "X-Sd-Publisher-Key: {{SDIO_PUBLISHER_KEY}}" \
-d '{"pairs": [{"Symbol":"AUDUSD", "Bid":0.74710, "Mid":0.74712, "Ask":0.74716},{"Symbol":"EURUSD", "Bid":0.74708, "Mid":0.74714, "Ask":0.74718}]}'
where {{SDIO_TOPIC_ID}} is the ID of your topic and {{SDIO_PUBLISHER_KEY}} the Publisher Key related to the app owning the topic.
  1. Use HTTP POST verb.

  2. Publish JSON content and set a Content-Type header to application/json in your request.

  3. Be aware a successfull response will be a HTTP 204 No Content.

9.4.1. Rate limit

X-RateLimit-period and X-RateLimit-Remaining-period response headers indicate how many publication requests can be made in a rolling time period (e.g Minute or Hour) and how many of those requests have already been made. If any of the limits is being reached, our server will return a HTTP 429 status code to the publisher with the following JSON body:

{ "message":"API rate limit exceeded" }

9.5. How to subscribe to a Streamdata.io Topic

  • In order to subscribe to your Streamdata.io Topic, open a terminal and run the following command:

curl "http://localhost:6080/v1/topics/{{SDIO_TOPIC_ID}}?X-Sd-Subscriber-Key={{SDIO_SUBSCRIBER_KEY}}"
where {{SDIO_TOPIC_ID}} is the ID of your topic and {{SDIO_PUBLISHER_KEY}} the Publisher Key related to the app owning the topic.
id: fec856bd-f22d-49fe-8226-7c70f2e74121
event: data
data: {"pairs":[{"Symbol":"AUDUSD","Bid":0.74709,"Mid":0.74713,"Ask":0.74717},
                {"Symbol":"EURUSD","Bid":0.74709,"Mid":0.74713,"Ask":0.74717}]}
  • Try to publish an updated version of the data, you should now receive a patch event containing a list of JSON Patch operations:

id: 19063830-e414-4df6-b08a-565f50a6e4d8
event: patch
data: [{"op":"replace","path":"/pairs/0/Bid","value":0.74710},
       {"op":"replace","path":"/pairs/0/Mid","value":0.74712},
       {"op":"replace","path":"/pairs/0/Ask","value":0.74716},
       {"op":"replace","path":"/pairs/1/Bid","value":0.74708},
       {"op":"replace","path":"/pairs/1/Mid","value":0.74714},
       {"op":"replace","path":"/pairs/1/Ask","value":0.74718}]
Refer to Response format for additional information about snapshot and patches.

10. Management API

The Streamdata.io Management API allows for programmatic access to the Streamdata.io Platform. You can efficiently manage your account, applications and associated topics. Our API is organized around REST and has predictable, resource-oriented URLs, and uses HTTP response codes to indicate API errors. We use built-in HTTP features, like HTTP verbs, which are understood by off-the-shelf HTTP clients. We support cross-origin resource sharing, allowing you to interact securely with our API from a client-side web application. JSON is returned by all API responses, including errors. Finally, we rely on gzip format to compress all API responses via the Content-Encoding entity header. Refer to our OpenAPI Specification for more details.

10.1. Authentication

Streamdata.io Management API uses JWT token to authenticate requests:

  1. First you need to create a user. If you do not have one yet, you can easily create it via a POST on /users endpoint.

  2. Once your account is created. Make a PUT request on /session endpoint to create a session object. The response will contain a valid JWT token in the token attribute that you must use in all subsequent requests to the Management API.

  3. To make calls to the API, send the previously retrieved token as a Authorization HTTP header using the Bearer authentication scheme.

10.2. Error Codes

Streamdata.io uses conventional HTTP response codes to indicate the success or failure of an API request. In general, codes in the 2xx range indicate success, codes in the 4xx range indicate an error that failed given the information provided (e.g., a required parameter was omitted, etc.), and codes in the 5xx range indicate an error with Streamdata.io’s servers.

10.2.1. HTTP status code summary

Status Code Description

200 - OK

Everything worked as expected.

400 - Bad Request

The request was unacceptable, often due to missing a required parameter.

401 - Unauthorized

No valid JWT token provided.

404 - Not Found

The requested resource doesn’t exist.

409 - Conflict

The request conflicts with another request.

429 - Too Many Requests

Too many requests hit the API too quickly. We recommend an exponential backoff of your requests.

500, 502, 503, 504 - Server Errors

Something went wrong on Streamdata.io’s end.

10.3. Pagination

To start with, it’s important to know a few facts about receiving paginated items:

  1. By default, a call to the Streamdata.io API provides items in sets of 100.

  2. You can specify how many items to receive (up to a maximum of 1000) via the per_page query parameter.

  3. All paginated queries start at page 0. It may sound counter-intuitive at first, but it makes life easier when it comes to building a UI pagination component.

Pagination information is provided in the links attribute of a response.

For example, let’s make a request to the /apps endpoint with the per_page query param set to 5.

The links attribute will contain an array of links pointing to the different pages allowing you to easily navigate.

{
  ...
  "links": [
    {
      "rel": "self",
      "href": "/v1/users/ed1ebfd2-bcd1-4b29-a3b3-ac266b5ab88d/apps?page=0&per_page=5"
    },
    {
      "rel": "first",
      "href": "/v1/users/ed1ebfd2-bcd1-4b29-a3b3-ac266b5ab88d/apps?page=0&per_page=5"
    },
    {
      "rel": "next",
      "href": "/v1/users/ed1ebfd2-bcd1-4b29-a3b3-ac266b5ab88d/apps?page=1&per_page=5"
    },
    {
      "rel": "last",
      "href": "/v1/users/ed1ebfd2-bcd1-4b29-a3b3-ac266b5ab88d/apps?page=3&per_page=5"
    }
  ]
}

10.3.1. Navigating through the pages

Now that you know how many pages there are to receive, you can start navigating through the pages to consume the results. You do this by passing in a page parameter.

10.3.2. Changing the number of items received

By passing the per_page parameter, you can specify how many items you want each page to return, up to 1000 items.

10.4. Versioning

We use Semantic Versioning. Given a version number MAJOR.MINOR.PATCH, we increment the:

  • MAJOR version when we make backwards-incompatible changes,

  • MINOR version when we add functionality in a backwards-compatible manner,

  • PATCH version when we make backwards-compatible bug fixes.

Streamdata.io API Management only requires the MAJOR version in the request URI. When we change the API in a backwards-incompatible way, we release a new MAJOR version. See section Backwards compatibility for more details.

Note that you can also choose to always request the latest version by using the keyword latest as the version number.

For some period of time, different versions of the same API can be able to work at the same time. This is to help the client smoothly transition from the older version to the newer version of the API.

Currently, the latest major supported version is v1.

10.4.1. Backwards compatibility

Backwards-compatible changes (non-breaking).
  • Adding an new API resources

  • Adding new optional request parameters to existing API methods

  • Adding a new method to an existing API resources

  • Adding a new optional field to a request body

  • Adding a new field to existing API responses

  • Changing the order of field in existing API responses

  • Adding a value to an enum

Backwards-incompatible changes (breaking).
  • Removing or renaming a service, interface, field, method or enum value

  • Changing an HTTP binding

  • Changing the type of a field

  • Changing a resource name format

  • Changing visible behavior of existing requests

  • Changing the URL format in the HTTP definition

We provide hypermedia links in the response contents so that the client can dynamically navigate to the appropriate resource by traversing the hypermedia links. This is conceptually the same as a web user navigating through web pages by clicking the appropriate hyperlinks in order to achieve a final goal.

For example, below an extract of the given JSON response that can come from an API like GET /app/{{appId}}:

{
  "id": "{{appId}}",
  "name": "MyApp",
  ...
  "links": [
    {
      "rel": "self",
      "href": "/v1/apps/{{appId}}"
    },
    {
      "rel": "topics",
      "href": "/v1/apps/{{appId}}/topics"
    },
    {
      "rel": "user",
      "href": "/v1/users/{{userId}}"
    }
  ]
}

10.6. Rate limit

X-RateLimit-period and X-RateLimit-Remaining-period response headers indicate how many requests a client can make in a rolling time period (e.g Minute or Hour) and how many of those requests the client has already spent. If any of the limits is being reached, our server will return a HTTP 429 status code to the client with the following JSON body:

{ "message":"API rate limit exceeded" }