Skip to content

Commit 2a90456

Browse files
author
betzrhodes
authored
Merge pull request #6 from electricimp/develop
Update CI
2 parents 083fec0 + ddaed64 commit 2a90456

File tree

5 files changed

+99
-89
lines changed

5 files changed

+99
-89
lines changed

Diff for: .editorconfig

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
root = true
2+
3+
[*]
4+
indent_style = space
5+
indent_size = 4
6+
end_of_line = lf
7+
charset = utf-8
8+
trim_trailing_whitespace = true
9+
insert_final_newline = true
10+
11+
[*.md]
12+
trim_trailing_whitespace = false

Diff for: .gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
.DS_Store
2+
.builder-cache

Diff for: .impt.test

+1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
{
22
"deviceGroupId": "ca83d15f-fcb8-d3c2-ff69-b0d4acb20e9b",
3+
"deviceGroupName" : "impFarm M",
34
"timeout": 180,
45
"stopOnFail": false,
56
"allowDisconnect": false,

Diff for: .travis.yml

-11
This file was deleted.

Diff for: README.md

+84-78
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ The AWSKinesisStreams library utilizes the [AWSRequestV4](https://github.com/ele
1616
#require "AWSKinesisStreams.agent.lib.nut:1.1.0"
1717
```
1818

19+
![Build Status](https://cse-ci.electricimp.com/app/rest/builds/buildType:(id:AWSKinesisStreams_BuildAndTest)/statusIcon)
20+
1921
## Example ##
2022

2123
A complete, step-by-step recipe can be found in the [Examples](./Examples) folder.
@@ -76,13 +78,13 @@ This class represents an AWS Kinesis Streams record: a combination of data attri
7678

7779
This method creates and returns an AWSKinesisStreams.Record object that can be written into an Amazon Kinesis stream using [AWSKinesisStreams.Producer](#awskinesisstreamsproducer-class) methods.
7880

79-
| Parameter | Data Type | Required? | Description |
81+
| Parameter | Data Type | Required | Description |
8082
| --- | --- | --- | --- |
8183
| *data* | Blob or [JSON-compatible type](#json-compatible-type) | Yes | The record data |
8284
| *partitionKey* | String | Yes | Identifies which shard in the stream the data record is assigned to (see the [Kinesis Streams documentation](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_Record.html#Streams-Type-Record-PartitionKey)) |
83-
| *explicitHashKey* | String | Optional | The hash value used to explicitly determine the shard the data record is assigned to by overriding the partition key hash (see the [Kinesis Streams documentation](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#Streams-PutRecord-request-ExplicitHashKey)) |
84-
| *prevSequenceNumber* | String | Optional | See the [Kinesis Streams documentation](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#Streams-PutRecord-request-SequenceNumberForOrdering) |
85-
| *encoder* | Function | Optional | A custom JSON encoder function for encoding the provided data (e.g. [JSONEncoder.encode](https://github.com/electricimp/JSONEncoder)) |
85+
| *explicitHashKey* | String | No | The hash value used to explicitly determine the shard the data record is assigned to by overriding the partition key hash (see the [Kinesis Streams documentation](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#Streams-PutRecord-request-ExplicitHashKey)) |
86+
| *prevSequenceNumber* | String | No | See the [Kinesis Streams documentation](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#Streams-PutRecord-request-SequenceNumberForOrdering) |
87+
| *encoder* | Function | No | A custom JSON encoder function for encoding the provided data (eg. [*JSONEncoder.encode()*](https://github.com/electricimp/JSONEncoder)) |
8688

8789
## AWS_KINESIS_STREAMS_ENCRYPTION_TYPE Enum ##
8890

@@ -103,7 +105,7 @@ This class allows the agent to write data records to a specific AWS Kinesis stre
103105

104106
Creates and returns an AWSKinesisStreams.Producer object. The constructor’s parameters are as follows:
105107

106-
| Parameter | Data Type | Required? | Description |
108+
| Parameter | Data Type | Required | Description |
107109
| --- | --- | --- | --- |
108110
| *region* | String | Yes | The Region code of Amazon EC2 (see the [Amazon EC2 documentation](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html).) |
109111
| *accessKeyId* | String | Yes | The access key ID of an AWS IAM user. See the [Kinesis Streams documentation](http://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-iam.html)) |
@@ -116,10 +118,10 @@ Creates and returns an AWSKinesisStreams.Producer object. The constructor’s pa
116118

117119
This method writes a single data record into the AWS Kinesis stream. For more information, please see the corresponding [Kinesis Streams REST API action](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html).
118120

119-
| Parameter | Data Type | Required? | Description |
121+
| Parameter | Data Type | Required | Description |
120122
| --- | --- | --- | --- |
121123
| *record* | [AWSKinesisStreams.Record](#awskinesisstreamsrecord-class) | Yes | The record to be written |
122-
| *callback* | Function | Optional | Executed once the operation is completed |
124+
| *callback* | Function | No | Executed once the operation is completed |
123125

124126
The method returns nothing. The result of the operation may be obtained via the *callback* function, which has the following parameters:
125127

@@ -132,10 +134,10 @@ The method returns nothing. The result of the operation may be obtained via the
132134

133135
This method writes multiple data records into the AWS Kinesis stream in a single request. Every record is processed by AWS individually. Some of the records may be written successfully but some may fail. For more information, please see the corresponding [Kinesis Streams REST API action](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html).
134136

135-
| Parameter | Data Type | Required? | Description |
137+
| Parameter | Data Type | Required | Description |
136138
| --- | --- | --- | --- |
137139
| *records* | Array of [AWSKinesisStreams.Records](#awskinesisstreamsrecord-class) | Yes | The records to be written |
138-
| *callback* | Function | Optional | Executed once the operation is completed |
140+
| *callback* | Function | No | Executed once the operation is completed |
139141

140142
The method returns nothing. The result of the operation may be obtained via the *callback* function, which has the following parameters:
141143

@@ -167,13 +169,13 @@ This class allows your code to read data records from a specific AWS Kinesis Str
167169

168170
This method creates and returns an AWSKinesisStreams.Consumer object. The constructor’s parameters are as follows:
169171

170-
| Parameter | Data Type | Required? | Description |
172+
| Parameter | Data Type | Required | Description |
171173
| --- | --- | --- | --- |
172174
| *region* | String | Yes | The Region code of Amazon EC2 (see the [EC2 documentation](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html)) |
173175
| *accessKeyId* | String | Yes | The access key ID of an AWS IAM user (see the [Kinesis Streams documentation](http://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-iam.html)) |
174176
| *secretAccessKey* | String | Yes | The secret access key of an AWS IAM user (see the [Kinesis Streams documentation](http://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-iam.html)) |
175177
| *streamName* | String | Yes | The name of an AWS Kinesis stream |
176-
| *isBlob* | Boolean | Optional | If `true`, the AWSKinesisStreams.Consumer object will consider every received data record as a Squirrel blob. If `false` or not specified, the AWSKinesisStreams.Consumer object will consider every received data record as a JSON data and parse it into an appropriate [JSON-compatible type](#json-compatible-type) |
178+
| *isBlob* | Boolean | No | If `true`, the AWSKinesisStreams.Consumer object will consider every received data record as a Squirrel blob. If `false` or not specified, the AWSKinesisStreams.Consumer object will consider every received data record as a JSON data and parse it into an appropriate [JSON-compatible type](#json-compatible-type) |
177179

178180
Before creating an AWSKinesisStreams.Consumer instance your code should know which type of data it is going to receive: binary data (a Squirrel blob) or a [JSON-compatible type](#json-compatible-type). This choice is specified in the AWSKinesisStreams.Consumer constructor and cannot be changed after that. In a complex case, your application can specify the data as a blob and parse it to a specific or custom type by itself.
179181

@@ -183,7 +185,7 @@ Before creating an AWSKinesisStreams.Consumer instance your code should know whi
183185

184186
This method retrieves a list of the IDs of all the shards in the AWS Kinesis stream, including closed shards. Closed shards may still contain records your application may need to read.
185187

186-
| Parameter | Data Type | Required? | Description |
188+
| Parameter | Data Type | Required | Description |
187189
| --- | --- | --- | --- |
188190
| *callback* | Function | Yes | Executed once the operation is completed |
189191

@@ -200,7 +202,7 @@ This method allows your code to specify a start position from which the reading
200202

201203
**Note** Every shard iterator returned by *getShardIterator()* or *getRecords()* expires five minutes after it is returned. Your application should call *getRecords()* with the iterator before it expires, otherwise the call will fail and your code will need to obtain a new iterator using *getShardIterator()*.
202204

203-
| Parameter | Data Type | Required? | Description |
205+
| Parameter | Data Type | Required | Description |
204206
| --- | --- | --- | --- |
205207
| *shardId* | String | Yes | The shard ID. |
206208
| *type* | [AWS_KINESIS_STREAMS_SHARD_ITERATOR_TYPE](#aws_kinesis_streams_shard_iterator_type-enum) | Yes | The shard iterator type. Determines how the shard iterator is used to start reading data records from the shard. Some of the types require the corresponding *typeOptions* to be specified |
@@ -223,12 +225,12 @@ The method returns nothing. The result of the operation may be obtained via the
223225

224226
This method allows your code to read a portion of data records using the specified shard iterator and returns the next shard iterator which can be used to read the next portion of data records by calling *getRecords()* again. Reading is always going to prefer older records over the latest. For more information, please see the corresponding [Kinesis Streams REST API action](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html).
225227

226-
| Parameter | Data Type | Required? | Description |
228+
| Parameter | Data Type | Required | Description |
227229
| --- | --- | --- | --- |
228230
| *options* | Table | Yes | Options for the operation (see below) |
229231
| *callback* | Function | Yes | Executed once the operation is completed |
230232

231-
| *options* key | Data Type | Required? | Description |
233+
| *options* key | Data Type | Required | Description |
232234
| --- | --- | --- | --- |
233235
| *shardIterator* | String | Yes | The shard iterator that specifies the position in the shard from which the reading should be started |
234236
| *limit* | Integer | Optional | The maximum number of data records to read. If not specified, the number of returned records is AWS Kinesis Streams specific (see the [Kinesis Streams documentation](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html#Streams-GetRecords-request-Limit)) |
@@ -276,8 +278,8 @@ A type of Squirrel data which can be encoded/decoded into/from JSON, eg. table,
276278
#require "AWSKinesisStreams.agent.lib.nut:1.1.0"
277279
#require "JSONEncoder.class.nut:2.0.0"
278280
279-
//This class can be used to hold numbers larger than Squirrel can natively support (i.e. anything larger than 32-bit)
280-
//and then be encoded as a number (rather than a string) when encoded with `JSONEncoder.encode`.
281+
// This class can be used to hold numbers larger than Squirrel can natively support (ie. anything larger than 32-bit)
282+
// and then be encoded as a number (rather than a string) when encoded with 'JSONEncoder.encode()'.
281283
class JSONLiteralString {
282284
_string = null;
283285
@@ -305,34 +307,36 @@ producer <- AWSKinesisStreams.Producer(AWS_KINESIS_REGION, AWS_KINESIS_ACCESS_KE
305307
306308
// Writes single data record
307309
producer.putRecord(AWSKinesisStreams.Record("Hello!", "partitionKey"), function (error, putResult) {
308-
if (error) {
309-
server.error("Data writing failed: " + error.details);
310-
} else {
311-
// Record written successfully
312-
}
310+
if (error) {
311+
server.error("Data writing failed: " + error.details);
312+
} else {
313+
// Record written successfully
314+
}
313315
});
314316
315317
// Writes multiple records with different data structures
316318
records <- [
317-
AWSKinesisStreams.Record("test", "partitionKey1"),
318-
AWSKinesisStreams.Record(12345, "partitionKey2"),
319-
AWSKinesisStreams.Record({ "temperature" : 21, "humidity" : 60 }, "partitionKey3"),
320-
AWSKinesisStreams.Record({ "a" : JSONLiteralString("123456789123456789") }, "partitionKey4", null, null, JSONEncoder.encode.bindenv(JSONEncoder)) //write record using custom encoder
319+
AWSKinesisStreams.Record("test", "partitionKey1"),
320+
AWSKinesisStreams.Record(12345, "partitionKey2"),
321+
AWSKinesisStreams.Record({ "temperature" : 21, "humidity" : 60 }, "partitionKey3"),
322+
323+
// Write record using custom encoder
324+
AWSKinesisStreams.Record({ "a" : JSONLiteralString("123456789123456789") }, "partitionKey4", null, null, JSONEncoder.encode.bindenv(JSONEncoder))
321325
];
322326
323327
producer.putRecords(records, function (error, failedRecordCount, putResults) {
324-
if (error) {
325-
server.error("Data writing failed: " + error.details);
326-
} else if (failedRecordCount > 0) {
327-
server.log("Data writing partially failed:");
328-
foreach (res in putResults) {
329-
if (res.errorCode) {
330-
server.log(format("%s: %s", res.errorCode, res.errorMessage));
331-
}
332-
}
333-
} else {
334-
// Records written successfully
328+
if (error) {
329+
server.error("Data writing failed: " + error.details);
330+
} else if (failedRecordCount > 0) {
331+
server.log("Data writing partially failed:");
332+
foreach (res in putResults) {
333+
if (res.errorCode) {
334+
server.log(format("%s: %s", res.errorCode, res.errorMessage));
335+
}
335336
}
337+
} else {
338+
// Records written successfully
339+
}
336340
});
337341
```
338342

@@ -353,60 +357,62 @@ consumer <- AWSKinesisStreams.Consumer(AWS_KINESIS_REGION, AWS_KINESIS_ACCESS_KE
353357
354358
// Obtains the stream shards
355359
consumer.getShards(function (error, shardIds) {
356-
if (error) {
357-
server.error("getShards failed: " + error.details);
358-
} else {
359-
foreach (shardId in shardIds) {
360-
getShardIterator(shardId);
361-
}
360+
if (error) {
361+
server.error("getShards failed: " + error.details);
362+
} else {
363+
foreach (shardId in shardIds) {
364+
getShardIterator(shardId);
362365
}
366+
}
363367
});
364368
365369
// Obtains shard iterator for the specified shard and starts reading records
366370
function getShardIterator(shardId) {
367-
consumer.getShardIterator(
368-
shardId,
369-
AWS_KINESIS_STREAMS_SHARD_ITERATOR_TYPE.TRIM_HORIZON,
370-
null,
371-
function (error, shardIterator) {
372-
if (error) {
373-
server.error("getShardIterator failed: " + error.details);
374-
} else {
375-
// shard iterator obtained successfully
376-
readRecords({ "shardIterator" : shardIterator, "limit" : 10 });
377-
}
378-
});
371+
consumer.getShardIterator(
372+
shardId,
373+
AWS_KINESIS_STREAMS_SHARD_ITERATOR_TYPE.TRIM_HORIZON,
374+
null,
375+
function (error, shardIterator) {
376+
if (error) {
377+
server.error("getShardIterator failed: " + error.details);
378+
} else {
379+
// shard iterator obtained successfully
380+
readRecords({ "shardIterator" : shardIterator, "limit" : 10 });
381+
}
382+
}
383+
);
379384
}
380385
381386
// Recursively reads records from the specified shard
382387
function readRecords(options) {
383-
consumer.getRecords(
384-
options,
385-
function (error, records, millisBehindLatest, nextOptions) {
386-
if (error) {
387-
server.error("Data reading failed: " + error.details);
388-
} else {
389-
if (records.len() == 0) {
390-
// No new records
391-
} else {
392-
foreach (record in records) {
393-
// Process records individually
394-
}
395-
}
396-
397-
if (nextOptions) {
398-
// Read next portion of records
399-
imp.wakeup(10.0, function () {
400-
readRecords(nextOptions);
401-
});
402-
}
403-
}
404-
});
388+
consumer.getRecords(
389+
options,
390+
function (error, records, millisBehindLatest, nextOptions) {
391+
if (error) {
392+
server.error("Data reading failed: " + error.details);
393+
} else {
394+
if (records.len() == 0) {
395+
// No new records
396+
} else {
397+
foreach (record in records) {
398+
// Process records individually
399+
}
400+
}
401+
402+
if (nextOptions) {
403+
// Read next portion of records
404+
imp.wakeup(10.0, function () {
405+
readRecords(nextOptions);
406+
});
407+
}
408+
}
409+
}
410+
);
405411
}
406412
```
407413

408414
Working examples are also provided in the [Examples](./Examples) directory and described [here](./Examples/README.md).
409415

410416
## License ##
411417

412-
The AWSKinesisStreams library is licensed under the [MIT License](./LICENSE).
418+
This library is licensed under the [MIT License](./LICENSE).

0 commit comments

Comments
 (0)