Merge pull request #64 from nobitlost/master

v2.8.0

Author: betzrhodes

Diff for: README.md

@@ -2,7 +2,7 @@
 
 _Builder_ combines a preprocessor with an expression language and advanced imports.
 
-**Current version: 2.7.0**
+**Current version: 2.8.0**
 
 ![Build Status](https://cse-ci.electricimp.com/app/rest/builds/buildType:(id:Builder_BuildAndTest)/statusIcon)
 
@@ -54,6 +54,7 @@ _Builder_ combines a preprocessor with an expression language and advanced impor
     - [Remote Includes](#remote-includes)
         - [Caching Remote Includes](#caching-remote-includes)
         - [Proxy Access To Remote Includes](#proxy-access-to-remote-includes)
+        - [Local Includes From Remote Files](#local-includes-from-remote-files)
 - [Testing](#testing)
 - [License](#license)
 
@@ -458,15 +459,15 @@ The following types are supported in expressions:
 Variables can be used in `Builder` expression evaluation.
 
 - Variables can be defined by:
-    - The `-D <variable name> <variable value>` command line parameter.
+    - The `-D<variable name> <variable value>` command line parameter.
     - A <code><b>@set</b></code> statement.
     - An [environment variable](#environment-variables).
 - Undefined variables are evaluated as `null`.
 - Variable names can contain `$`, `_`, latin letters and digits. They must not start with a digit.
 
 #### Variable Definition Order ####
 
-1. When resolving a variable’s value, *Builder* first looks for its definition in the command line `-D` parameters (`-D <variable name> <variable value>`) passed to the *pleasebuild* command.
+1. When resolving a variable’s value, *Builder* first looks for its definition in the command line `-D` parameters (`-D<variable name> <variable value>`) passed to the *pleasebuild* command.
 1. If no such variable definition is found, Squirrel code is scanned for `@set` statements preceding the variable usage.
 1. If no variable definitions are found in the previous steps, *Builder* looks in the host environment variables.
 
@@ -592,9 +593,10 @@ npm install -g Builder
 then use the `pleasebuild` command which is provided by Builder:
 
 ```
-pleasebuild [-l] [-D <variable> <value>]
+pleasebuild [-l] [-D<variable> <value>]
     [--github-user <username> --github-token <token>]
-    [--lib <path_to_file>] [--suppress-duplicate-includes-warning]
+    [--lib <path_to_file>]
+    [--use-remote-relative-includes] [--suppress-duplicate-includes-warning]
     [--cache] [--clear-cache] [--cache-exclude-list <path_to_file>]
     [--save-dependencies [<path_to_file>]] [--use-dependencies [<path_to_file>]]
     [--save-directives [<path_to_file>]] [--use-directives [<path_to_file>]]
@@ -610,10 +612,11 @@ and the options are:
 | Option | Synonym | Mandatory? | Value&nbsp;Required? | Description |
 | --- | --- | --- | --- | --- |
 | -l |  | No | No | Generates line control statements. For a more detailed explanation, please read [this GCC page](https://gcc.gnu.org/onlinedocs/gcc-4.5.4/cpp/Line-Control.html) |
-| -D &lt;variable&gt; | | No | Yes | Defines a [variable](#variables). May be specified several times to define multiple variables |
+| -D&lt;variable&gt; | | No | Yes | Defines a [variable](#variables). May be specified several times to define multiple variables |
 | --github-user | | No | Yes | A GitHub username. See [‘Files From GitHub’](#files-from-github) |
 | --github-token | | No | Yes | A GitHub [personal access token](https://github.com/settings/tokens) or password (not recommended). Should be specified if the `--github-user` option is specified. See [‘Files From GitHub’](#files-from-github) |
 | --lib | --libs | No | Yes | Include the specified [JavaScript file(s) as a library](#including-javascript-libraries). May be specified several times to include multiple libraries. The provided value may specify a concrete file or a directory (all files from the directory will be included). The value may contain [wildcards](https://www.npmjs.com/package/glob) (all matched files will be included) |
+| --use-remote-relative-includes | | No | No | Interpret every [local include](#local-files) as relative to the location of the source file where it is mentioned. See ['Local Includes From Remote Files'](#local-includes-from-remote-files) |
 | --suppress-duplicate-includes-warning | --suppress-duplicate | No | No | Do not show a warning if a source file with the same content was included multiple times from different locations and this results in code duplication |
 | --cache | -c | No | No | Turn on caching for all files included from remote resources. This option is ignored if the `--save-dependencies` or `--use-dependencies` options are specified. See [‘Caching Remote Includes’](#caching-remote-includes) |
 | --clear-cache | | No | No | Clear the cache before Builder starts running. See [‘Caching Remote Includes’](#caching-remote-includes) |
@@ -637,32 +640,36 @@ then instantiate, setup and execute Builder from the source code, for example:
 const Builder = require('Builder');
 const builder = new Builder();
 
-// Specify whether you need line control statements. See the "-l" CLI option
+// Specify whether you need line control statements. See the "-l" CLI option.
 builder.machine.generateLineControlStatements = <true|false>;
 
-// Cache all files included from remote sources. See the "--cache" CLI option
+// Cache all files included from remote sources. See the "--cache" CLI option.
 builder.machine.useCache = <true|false>;
 
-// Set GitHub credentials. See the "--github-user" and "--github-token" CLI options
+// Set GitHub credentials. See the "--github-user" and "--github-token" CLI options.
 builder.machine.readers.github.username = "<USERNAME>";
 builder.machine.readers.github.token = "<PASSWORD_OR_ACCESS_TOKEN>";
 
 // Path to the file that lists the resources which should be excluded from caching.
-// See the "--cache-exclude-list" CLI option
+// See the "--cache-exclude-list" CLI option.
 builder.machine.excludeList = "<PATH_TO_FILE>";
 
+// Replace local include paths to github URLs if requested.
+// See the "--use-remote-relative-includes" CLI option.
+builder.machine.remoteRelativeIncludes = <true|false>;
+
 // Suppress warning about duplicate includes.
-// See the "--suppress-duplicate-includes-warning" CLI option
+// See the "--suppress-duplicate-includes-warning" CLI option.
 builder.machine.suppressDupWarning = <true|false>;
 
-// See the "--save-dependencies" CLI option
+// See the "--save-dependencies" CLI option.
 builder.machine.dependenciesSaveFile = <false|"PATH_TO_FILE">;
-// See the "--use-dependencies" CLI option
+// See the "--use-dependencies" CLI option.
 builder.machine.dependenciesUseFile = <false|"PATH_TO_FILE">;
 
-// See the "--save-directives" CLI option
+// See the "--save-directives" CLI option.
 builder.machine.directivesSaveFile = <false|"PATH_TO_FILE">;
-// See the "--use-directives" CLI option
+// See the "--use-directives" CLI option.
 builder.machine.directivesUseFile = <false|"PATH_TO_FILE">;
 
 const inputFile = "PATH_TO_YOUR_INPUT_FILE";
@@ -675,11 +682,11 @@ To understand Builder setup, please review [this source code](./src/cli.js).
 
 ## Reproducible Artifacts ##
 
-It is possible to save the build configuration used for preprocessing a source file &mdash; references to the concrete versions of GitHub files and libraries that are used, and to Builder variable definitions &mdash; and preprocess the source file again later with the saved configuration.
+It is possible to save the build configuration used for preprocessing a source file &mdash; references to the concrete versions of GitHub files and libraries that are used, and Builder variable definitions which are used &mdash; and preprocess the source file again later with the saved configuration.
 
 ### GitHub Files: Dependencies ###
 
-`--save-dependencies [<path_to_file>]` and `--use-dependencies [<path_to_file>]` options are used to save and to reuse, respectively, references to concrete versions of GitHub files and libraries. The references are saved in a JSON file. If a file is not specified, Builder will attempt to read a `dependencies.json` file from the local directory. Every reference consists of GitHub file URL and Git Blob ID (Git Blob SHA). For more information, please see [the Git Manual](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects) and [the GitA API](https://developer.github.com/v3/git/blobs/).
+`--save-dependencies [<path_to_file>]` and `--use-dependencies [<path_to_file>]` options are used to save and to reuse, respectively, references to concrete versions of GitHub files and libraries. The references are saved in a JSON file. If a file name is not specified, the `dependencies.json` file in the local directory is used. Every reference consists of GitHub file URL and Git Blob ID (Git Blob SHA). For more information, please see [the Git Manual](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects) and [the Git API](https://developer.github.com/v3/git/blobs/).
 
 **Note** It is possible to obtain the Git Blob ID of a GitHub file using the following *git* command: `git hash-object <path_to_file>`
 
@@ -695,14 +702,6 @@ These options are processed the following way:
 
 **Note** If either `--save-dependencies` or `--use-dependencies` is specified, the `--cache` option is ignored.
 
-### Builder Variables: Directives ###
-
-The `--save-directives [<path_to_file>]` and `--use-directives [<path_to_file>]` options are used to, respectively, save and reuse Builder variable definitions. The definitions are saved in a JSON file. If a file is not specified, Builder will attempt to read a `directives.json` file from the local directory. These options are processed the similar way as the `--save-dependencies` and `--use-dependencies` options, above.
-
-When the `--use-directives [<path_to_file>]` option is used, the saved Builder variable definitions are merged with definitions specified by `-D <variable> <value>` options.
-
-### Example Files ###
-
 A typical `dependencies.json` file looks like this:
 
 ```json
@@ -718,6 +717,12 @@ A typical `dependencies.json` file looks like this:
 ]
 ```
 
+### Builder Variables: Directives ###
+
+The `--save-directives [<path_to_file>]` and `--use-directives [<path_to_file>]` options are used to, respectively, save and reuse Builder variable definitions. The definitions are saved in a JSON file. If a file is not specified, the `directives.json` file in the local directory is used. These options are processed the similar way as the `--save-dependencies` and `--use-dependencies` options, above.
+
+When the `--use-directives [<path_to_file>]` option is used, the saved Builder variable definitions are merged with definitions specified by `-D<variable> <value>` options.
+
 A typical `directives.json` file looks like this:
 
 ```json
@@ -843,6 +848,16 @@ For example, to operate through a proxy running at IP address 192.168.10.2 on po
 
 **Note** Files retrieved from GitHub (`github:` protocol) are always accessed using HTTPS. So when specifying a proxy in this case, make sure you use set the `HTTPS_PROXY` environment variable.
 
+### Local Includes From Remote Files ###
+
+By default, all [local includes](#local-files), even if they are mentioned in remote source files, are always interpreted as relative to the system where Builder is running.
+
+If `--use-remote-relative-includes` option is specified, every [local include](#local-files) is interpreted as relative to the location of the source file where it is mentioned. For example, a local include mentioned in remote source file from GitHub will be downloaded from the same GitHub URL as the source file.
+
+`--use-remote-relative-includes` option does not affect includes with [absolute remote paths](#remote-files).
+
+**Note** In the current Builder version `--use-remote-relative-includes` option affects includes mentioned in remote source files from GitHub only.
+
 # Testing #
 
 ```sh

Diff for: package.json

@@ -1,6 +1,6 @@
 {
   "name": "Builder",
-  "version": "2.7.0",
+  "version": "2.8.0",
   "description": "Builder Language Implementation",
   "main": "src/index.js",
   "bin": {

Diff for: spec/Machine/remote-relative-includes.spec.js

@@ -0,0 +1,35 @@
+// Copyright (c) 2016-2019 Electric Imp
+// This file is licensed under the MIT License
+// http://opensource.org/licenses/MIT
+
+'use strict';
+
+require('jasmine-expect');
+
+const init = require('./init')('main');
+const eol = require('eol');
+
+const githubPathA = "github:electricimp/Builder/spec/fixtures/sample-1/inc-a.nut"
+
+describe('Machine', () => {
+  let machine;
+
+  beforeEach(() => {
+    machine = init.createMachine();
+  });
+
+  it('check that locally included file will be checkouted from github', () => {
+    const fileNotFoundMessage = 'Local file "inc-b.nut" not found (github:electricimp/Builder/spec/fixtures/sample-1/inc-a.nut:2)';
+    try {
+      eol.lf(machine.execute(`@include once "${githubPathA}"`));
+      fail();
+    } catch (e) {
+      expect(e.message).toEqual(fileNotFoundMessage);
+    }
+
+    // check respect-local-includes feature
+    machine.remoteRelativeIncludes = true;
+    const res = eol.lf(machine.execute(`@include once "${githubPathA}"`));
+    expect(res).toEqual(`// included file a\n// included file b\n`);
+  });
+});

Diff for: src/Machine.js

@@ -283,6 +283,36 @@ class Machine {
     }
   }
 
+  /**
+   * Replace local includes to github URLs if requested
+   * @param {string} includePath
+   * @param {{}} context
+   * @private
+   */
+  _remoteRelativeIncludes(includePath, context) {
+    if (!this.remoteRelativeIncludes) {
+      return includePath;
+    }
+
+    // check if the file is included locally
+    if (this._getReader(includePath) !== this.readers.file) {
+      return includePath;
+    }
+
+    // check that path is not absolute
+    if (path.isAbsolute(includePath)) {
+      return includePath;
+    }
+
+    // check if file is included from github source
+    const remotePath = this._formatPath(context.__PATH__, includePath);
+    if (this._getReader(remotePath) === this.readers.github) {
+      return remotePath;
+    }
+
+    return includePath;
+  }
+
   /**
    * Include source
    * @param {string} source
@@ -295,7 +325,7 @@ class Machine {
   _includeSource(source, context, buffer, once, evaluated) {
 
     // path is an expression, evaluate it
-    const includePath = evaluated ? source : this.expression.evaluate(
+    let includePath = evaluated ? source : this.expression.evaluate(
       source,
       context,
     ).trim();
@@ -306,6 +336,9 @@ class Machine {
       return;
     }
 
+    // checkout local includes in the github sources from github 
+    includePath = this._remoteRelativeIncludes(includePath, context);
+
     const reader = this._getReader(includePath);
     this.logger.info(`Including source "${includePath}"`);
 

Diff for: src/cli.js

@@ -62,25 +62,26 @@ function usageInfo() {
 
 usage:\n\t\u001b[34m${Object.getOwnPropertyNames((packageJson.bin))[0]} [-l] [-D<variable> <value>]
 \t\t[--github-user <username> --github-token <token>]
-\t\t[--lib <path_to_file>] [--suppress-duplicate-includes-warning]
+\t\t[--lib <path_to_file>] [--use-remote-relative-includes] [--suppress-duplicate-includes-warning]
 \t\t[--cache] [--clear-cache] [--cache-exclude-list <path_to_file>]
 \t\t[--save-dependencies [<path_to_file>]] [--use-dependencies [<path_to_file>]]
 \t\t[--save-directives [<path_to_file>]] [--use-directives [<path_to_file>]] <input_file>\u001b[39m
 
 where:
 \t\u001b[34m-l\u001b[39m - generates line control statements
-\t\u001b[34m-D<varname> <value>\u001b[39m - define a variable that will be available from the source
-\t\u001b[34m--github-user <username>\u001b[39m - username for GitHub
-\t\u001b[34m--github-token <token>\u001b[39m - personal access token or password for GitHub
-\t\u001b[34m--lib <path_to_file>\u001b[39m - includes the specified JavaScript file(s) as a library
-\t\u001b[34m--suppress-duplicate-includes-warning\u001b[39m - does not show a warning if a source file with the exact content was included multiple times
-\t\u001b[34m--cache>\u001b[39m - turns on cache for all files included from remote resources
-\t\u001b[34m--clear-cache\u001b[39m - clears cache before Builder starts running
-\t\u001b[34m--cache-exclude-list <path_to_file>\u001b[39m - path to the file that lists the resources which should be excluded from caching
-\t\u001b[34m--save-dependencies [path_to_file]\u001b[39m - saves references to the used versions of GitHub files in the specified file
-\t\u001b[34m--use-dependencies [path_to_file]\u001b[39m - reads from the specified file references to the versions of GitHub files which should be used
-\t\u001b[34m--save-directives [path_to_file]\u001b[39m - saves Builder variable definitions in the specified file
-\t\u001b[34m--use-directives [path_to_file]\u001b[39m - reads from the specified file Builder variable definitions which should be used
+\t\u001b[34m-D<varname> <value>\u001b[39m - defines a variable
+\t\u001b[34m--github-user <username>\u001b[39m - a GitHub username
+\t\u001b[34m--github-token <token>\u001b[39m - a GitHub personal access token or password
+\t\u001b[34m--lib <path_to_file>\u001b[39m - include the specified JavaScript file(s) as a library
+\t\u001b[34m--use-remote-relative-includes\u001b[39m - interpret every local include as relative to the location of the source file where it is mentioned
+\t\u001b[34m--suppress-duplicate-includes-warning\u001b[39m - do not show a warning if a source file with the same content was included multiple times
+\t\u001b[34m--cache>\u001b[39m - turn on caching for all files included from remote resources
+\t\u001b[34m--clear-cache\u001b[39m - clear the cache before Builder starts running
+\t\u001b[34m--cache-exclude-list <path_to_file>\u001b[39m - set the path to the file that lists resources which should not be cached
+\t\u001b[34m--save-dependencies [path_to_file]\u001b[39m - save references to the required GitHub files in the specified file
+\t\u001b[34m--use-dependencies [path_to_file]\u001b[39m - use the specified file to set which GitHub files are required
+\t\u001b[34m--save-directives [path_to_file]\u001b[39m - save Builder variable definitions in the specified file
+\t\u001b[34m--use-directives [path_to_file]\u001b[39m - use Builder variable definitions from the specified file
 \t\u001b[34m<input_file>\u001b[39m — is the path to source file which should be preprocessed
     `.trim());
 }
@@ -163,6 +164,8 @@ function readArgs() {
         throw Error('Expected argument value after ' + argument);
       }
       res.directivesSaveFile = getOption(args, directivesDefaultFileName);
+    } else if ('--use-remote-relative-includes' === argument) {
+      res.remoteRelativeIncludes = true;
     } else if ('--suppress-duplicate-includes-warning' === argument || '--suppress-duplicate' === argument) {
       res.suppressDupWarning = true;
     } else if ('--use-dependencies' === argument) {
@@ -209,6 +212,8 @@ try {
   builder.machine.readers.github.token = args.gh.token;
   //set cache settings
   builder.machine.excludeList = args.excludeFile;
+  // set remote relative includes
+  builder.machine.remoteRelativeIncludes = args.remoteRelativeIncludes;
   // set supress dupicate includes warning
   builder.machine.suppressDupWarning = args.suppressDupWarning;
   // use dependencies