Skip to content

Instantly share code, notes, and snippets.

@herrphon
Forked from Paraphraser/MQTT_Node-Red_Influx.md
Created October 7, 2023 21:21
Show Gist options
  • Save herrphon/7a1121e16471cfe66ad20a8e430b6c00 to your computer and use it in GitHub Desktop.
Save herrphon/7a1121e16471cfe66ad20a8e430b6c00 to your computer and use it in GitHub Desktop.

Revisions

  1. @Paraphraser Paraphraser revised this gist Aug 3, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -61,7 +61,7 @@ The second command installs the MQTT broker. You will only need this if you do n

    ### InfluxDB

    If your copy of *InfluxDB* is running inside a *Docker* container, consider adding this *alias* statement to your .profile:
    If your copy of *InfluxDB* is running inside a *Docker* container, consider adding this *alias* statement to your .profile or .bashrc:

    ```
    $ alias influx='docker exec -it influxdb influx -precision=rfc3339'
  2. @Paraphraser Paraphraser revised this gist Sep 3, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -261,7 +261,7 @@ Click the Deploy button near the top, right of the canvas.

    Add two "debug" nodes to the canvas. Double-click each in turn and set its Output to "complete msg object". Connect the outlet of the "mqtt in" node to the first "debug" node, and the outlet of the "change" node to the second "debug" node. The final result should look something like this:

    ![Node-Red test flow](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-test-flow.png)
    ![Node-Red test flow](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/a43181ad22cbf8a3f0630a311fe80a1e0a456e3b/mqtt-node-red-influx-test-flow.png)

    Select the Debug panel (the controls below "Deploy").

  3. @Paraphraser Paraphraser revised this gist Sep 3, 2022. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -98,7 +98,7 @@ Back in the *Node-Red* main window, click the "+" button to add a new empty flow

    Drag an "mqtt in" node onto the canvas. Double-click to open and configure as follows:

    ![mqtt-in-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-mqtt-in-node.png)
    ![mqtt-in-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/0e559e4ee97697ed7dba14aba27f17d8a1df6526/mqtt-node-red-influx-mqtt-in-node.png)

    1. Select the option to add a new mqtt-broker.
    2. Click the pencil icon to open the server properties panel.
    @@ -261,7 +261,7 @@ Click the Deploy button near the top, right of the canvas.

    Add two "debug" nodes to the canvas. Double-click each in turn and set its Output to "complete msg object". Connect the outlet of the "mqtt in" node to the first "debug" node, and the outlet of the "change" node to the second "debug" node. The final result should look something like this:

    ![Node-Red test flow](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/0e559e4ee97697ed7dba14aba27f17d8a1df6526/mqtt-node-red-influx-mqtt-in-node.png)
    ![Node-Red test flow](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-test-flow.png)

    Select the Debug panel (the controls below "Deploy").

  4. @Paraphraser Paraphraser revised this gist Sep 3, 2022. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -109,7 +109,7 @@ Drag an "mqtt in" node onto the canvas. Double-click to open and configure as fo
    * If *Node-Red* and *Mosquitto* are running on different hosts then this will be a static IP address or the fully-qualified domain name of the host running *Mosquitto*.
    5. Click "Add".
    6. Enter the topic string ("/site/topic").
    7. Set the "Output" popup to "a parsed JSON object").
    7. Set the "Output" popup to "a parsed JSON object".
    8. Enter a name for the node. This appears in the schematic and it is good practice to repeat the topic string.
    9. Click "Done" to complete the node setup.

    @@ -140,7 +140,7 @@ Connect the outlet of the "mqtt in" node to the inlet of the "change" node.

    Double-click the "change" node to open and configure as follows:

    ![change-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-change-node.png)
    ![change-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/0e559e4ee97697ed7dba14aba27f17d8a1df6526/mqtt-node-red-influx-change-node.png)

    1. Enter a name for the node. This appears in the schematic and it is good practice to summarise the purpose of the node.
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
    @@ -225,7 +225,7 @@ Connect the outlet of the "change" node to the inlet of the "influxdb out" node.

    Double-click the "influxdb out" node to open and configure as follows:

    ![influxdb-out-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-influxdb-out-node.png)
    ![influxdb-out-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/0e559e4ee97697ed7dba14aba27f17d8a1df6526/mqtt-node-red-influx-influxdb-out-node.png)

    1. From the "Server" popup menu, choose "Add new influxdb...".
    2. Click the pencil icon to open the server properties panel.
    @@ -261,7 +261,7 @@ Click the Deploy button near the top, right of the canvas.

    Add two "debug" nodes to the canvas. Double-click each in turn and set its Output to "complete msg object". Connect the outlet of the "mqtt in" node to the first "debug" node, and the outlet of the "change" node to the second "debug" node. The final result should look something like this:

    ![Node-Red test flow](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-test-flow.png)
    ![Node-Red test flow](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/0e559e4ee97697ed7dba14aba27f17d8a1df6526/mqtt-node-red-influx-mqtt-in-node.png)

    Select the Debug panel (the controls below "Deploy").

  5. @Paraphraser Paraphraser revised this gist Jan 2, 2021. 1 changed file with 16 additions and 15 deletions.
    31 changes: 16 additions & 15 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -146,23 +146,22 @@ Double-click the "change" node to open and configure as follows:
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
    3. Click the ellipsis ("…") to open the expression editor.
    4. Copy the expression below and paste it into this window.
    5. Click "Done".
    6. Click "Done" to complete the node setup.

    The expression at step 4 is:
    ```
    [
    {
    "flag": msg.payload.b,
    "discrete": msg.payload.i,
    "continuous": msg.payload.r,
    "message": msg.payload.s
    },{
    "identity": msg.payload.t
    }
    ]
    ```

    ```
    [
    {
    "flag": msg.payload.b,
    "discrete": msg.payload.i,
    "continuous": msg.payload.r,
    "message": msg.payload.s
    },{
    "identity": msg.payload.t
    }
    ]
    ```
    5. Click "Done".
    6. Click "Done" to complete the node setup.

    The purpose of this node is to provide a cross-walk between the JSON keys ("b", "i", "r", "s" and "t"), and the field and tag names you need in the *InfluxDB* database. The basic pattern is:

    @@ -243,6 +242,8 @@ Double-click the "influxdb out" node to open and configure as follows:

    All other fields can be left at their default settings or changed to suit your requirements.

    > Warning: InfluxDB database connections are **global** to Node-Red. Suppose you have an existing flow connected to the "test" database. When you create a new flow for a new database, it is very tempting to copy the "influxdb out" node from the old flow, paste it into the new flow, open it, click the pencil icon, and just change the database name. If you do that, you will break your old flow because it will refer to the new database. **Always** start from scratch by dragging a new "influxdb out" node onto the canvas.
    Given the output from the "change" node, the practical effect of this node is:

    ```
  6. @Paraphraser Paraphraser revised this gist Oct 25, 2020. 1 changed file with 12 additions and 11 deletions.
    23 changes: 12 additions & 11 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -13,7 +13,6 @@ This recipe shows you how to get an MQTT payload into an *InfluxDB* database usi

    This recipe also show how to map between JSON keys and database field and tag names.


    ## Task goal

    Given this MQTT message structure:
    @@ -74,6 +73,8 @@ That alias allows you to connect to the "influx" command line interface (CLI) si
    $ influx
    ```

    > By default, "influx" displays time as nanoseconds since 1970-01-01 UTC. The `-precision=rfc3339` argument displays time in human-readable form.
    The "test" database must be created by hand. If you omit this step you will get an error from *Node-Red*. You can initialise the database like this:

    ```
    @@ -153,12 +154,12 @@ The expression at step 4 is:
    ```
    [
    {
    "flag": payload.b,
    "discrete": payload.i,
    "continuous": payload.r,
    "message": payload.s
    "flag": msg.payload.b,
    "discrete": msg.payload.i,
    "continuous": msg.payload.r,
    "message": msg.payload.s
    },{
    "identity": payload.t
    "identity": msg.payload.t
    }
    ]
    ```
    @@ -168,10 +169,10 @@ The purpose of this node is to provide a cross-walk between the JSON keys ("b",
    ```
    [
    {
    fieldName : payload.key,
    fieldName : msg.payload.key,
    ...
    },{
    tagName : payload.key,
    tagName : msg.payload.key,
    ...
    }
    ]
    @@ -181,7 +182,7 @@ If you only want to pass fields, then omit the square brackets and the elements

    ```
    {
    fieldName : payload.key,
    fieldName : msg.payload.key,
    ...
    }
    ```
    @@ -261,10 +262,10 @@ Add two "debug" nodes to the canvas. Double-click each in turn and set its Outpu

    ![Node-Red test flow](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-test-flow.png)

    Click "Deploy" to activate.

    Select the Debug panel (the controls below "Deploy").

    Click "Deploy" to activate. Any errors will show up in the Debug panel.

    Copy the following text to the clipboard then paste it into a text editor.

    ```
  7. @Paraphraser Paraphraser revised this gist Jul 8, 2020. 1 changed file with 19 additions and 5 deletions.
    24 changes: 19 additions & 5 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -85,7 +85,9 @@ $ influx
    ### Node-Red

    * Launch your browser and connect to your *Node-Red* server.
    * Use the main menu (three horizontal bars "≡" at the top, right of the *Node-Red* window) to open the [Palette Manager](https://nodered.org/docs/user-guide/runtime/adding-nodes), select the "Install" tab, then search for and install [node-red-contrib-influxdb](https://flows.nodered.org/node/node-red-contrib-influxdb). If you prefer to install contributions from the command line, do that.
    * Use the main menu (three horizontal bars "≡" at the top, right of the *Node-Red* window) to open the [Palette Manager](https://nodered.org/docs/user-guide/runtime/adding-nodes):
    * select the "Nodes" tab and check whether `node-red-contrib-influxdb` is already installed. If it is **not** installed,
    * select the "Install" tab, then search for and install [node-red-contrib-influxdb](https://flows.nodered.org/node/node-red-contrib-influxdb). If you prefer to install contributions from the command line, do that.

    ## A three-node flow

    @@ -105,7 +107,7 @@ Drag an "mqtt in" node onto the canvas. Double-click to open and configure as fo
    * In a non-*Docker* enviroment where *Node-Red* and *Mosquitto* are running on the same host, this will be the loopback address 127.0.0.1.
    * If *Node-Red* and *Mosquitto* are running on different hosts then this will be a static IP address or the fully-qualified domain name of the host running *Mosquitto*.
    5. Click "Add".
    6. Enter a topic (eg "/site/topic").
    6. Enter the topic string ("/site/topic").
    7. Set the "Output" popup to "a parsed JSON object").
    8. Enter a name for the node. This appears in the schematic and it is good practice to repeat the topic string.
    9. Click "Done" to complete the node setup.
    @@ -171,7 +173,7 @@ The purpose of this node is to provide a cross-walk between the JSON keys ("b",
    },{
    tagName : payload.key,
    ...
    },
    }
    ]
    ```

    @@ -184,9 +186,21 @@ If you only want to pass fields, then omit the square brackets and the elements
    }
    ```

    Note that it _is_ feasible to omit this "change" node entirely. If you do that the JSON keys in the *Node-Red* "payload" variable will become the field names in the database. But taking that shortcut carries some risks. For example, if an MQTT client starts providing unexpected JSON keys, those can easily pollute your *InfluxDB* database. Using a cross-walk between the expected JSON keys and the field and tag names you want in the database provides an effective barrier against unexpected keys.
    Note that it _is_ feasible to omit this "change" node entirely. If you do that the JSON keys in the *Node-Red* "payload" variable will become the field names in the database. Before you take that shortcut, consider:

    * If an MQTT client starts providing unexpected JSON keys, those can easily pollute your *InfluxDB* database. Using a cross-walk between the expected JSON keys and the field and tag names you want in your database provides an effective barrier against unexpected keys.
    * You may wish to include keys in your MQTT payload that you do **not** want to wind up in your database. Good examples are millis() uptime, free heap and version numbers. Such values are usually ephemeral and only of interest at the moment when they are produced (and might only be produced when you are actively debugging your MQTT client). You can always see such values by subscribing to the MQTT feed or attaching a Debug node to the "mqtt in" node.
    * A "change" node simplifies the process of adding new tags and fields. You can:
    * Add a new key+value pair to the JSON payload being produced by your MQTT client, then
    * Attach a Debug node to the "mqtt in" node to confirm that you are receiving the expected data, then
    * Change the cross-walk when you are ready to start importing the data into your database.
    * If you want to include both tags and fields in your database, you really only have two options, either:
    * your MQTT client has to format the JSON payload correctly before transmission, or
    * you need a "change" node to implement a cross-walk.

    * Opting to do the work in your MQTT client effectively rules out the tactical use of ephemeral values and a *step-wise refinement* approach to development if you need to add new fields.

    Given the output from the "mqtt in" node, this Javascript expression in the "change" node will result in:
    Given the output from the "mqtt in" node, the Javascript expression in the "change" node will result in:

    ```
    msg.payload = [
  8. @Paraphraser Paraphraser revised this gist Mar 2, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -218,7 +218,7 @@ Double-click the "influxdb out" node to open and configure as follows:
    3. Supply the network path to the host running *InfluxDB*:
    * In a *Docker* environment, this will be the name of the container running *InfluxDB* (eg "influxdb").
    * In a non-*Docker* enviroment where *Node-Red* and *InfluxDB* are running on the same host, this will be the loopback address 127.0.0.1.
    * If *Node-Red* and *InfluxDB* are running on different hosts then this will be the be a static IP address or the fully-qualified domain name of the host running *InfluxDB*.
    * If *Node-Red* and *InfluxDB* are running on different hosts then this will be a static IP address or the fully-qualified domain name of the host running *InfluxDB*.
    4. The name of the *InfluxDB* database. This needs to be the same as you created earlier ("CREATE DATABASE test").
    5. I recommend leaving this field blank. If you do then the "Server" field in the previous panel will take on the "host:port/database" appearance shown at the end of the dotted line in the figure. You lose that valuable synopsis by supplying a name in this field.
    6. Click "Add".
  9. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 21 additions and 21 deletions.
    42 changes: 21 additions & 21 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -165,47 +165,47 @@ The purpose of this node is to provide a cross-walk between the JSON keys ("b",

    ```
    [
    {
    fieldName : payload.key,
    ...
    },{
    tagName : payload.key,
    ...
    },
    {
    fieldName : payload.key,
    ...
    },{
    tagName : payload.key,
    ...
    },
    ]
    ```

    If you only want to pass fields, then omit the square brackets and the elements that describe the tag(s), like this:

    ```
    {
    fieldName : payload.key,
    ...
    fieldName : payload.key,
    ...
    }
    ```

    Note that it _is_ feasible to omit this "change" node entirely. If you do that the JSON keys in the *Node-Red* "payload" variable will become the field names in the database. But taking that shortcut carries some risks. For example, if an MQTT client starts providing unexpected JSON keys, those can easily pollute your *InfluxDB* database. Using a cross-walk between the expected JSON keys and the field and tag names you want in the database provides an effecive barrier against unexpected keys.
    Note that it _is_ feasible to omit this "change" node entirely. If you do that the JSON keys in the *Node-Red* "payload" variable will become the field names in the database. But taking that shortcut carries some risks. For example, if an MQTT client starts providing unexpected JSON keys, those can easily pollute your *InfluxDB* database. Using a cross-walk between the expected JSON keys and the field and tag names you want in the database provides an effective barrier against unexpected keys.

    Given the output from the "mqtt in" node, this Javascript expression in the "change" node will result in:

    ```
    msg.payload = [
    {
    flag: true,
    discrete: 123,
    continuous: 456.78,
    message: "hello world"
    },{
    identity: "tagValue"
    }
    ]
    {
    flag: true,
    discrete: 123,
    continuous: 456.78,
    message: "hello world"
    },{
    identity: "tagValue"
    }
    ]
    ```

    ### 3. "influxdb out" node

    Drag an "influxdb out" node onto the canvas.

    > Can't find the "influxdb out" node in the palette? Double-check that you "installed node-red-contrib-influxdb" as described above.
    > Can't find the "influxdb out" node in the palette? Double-check that you installed "node-red-contrib-influxdb" as described above.
    Connect the outlet of the "change" node to the inlet of the "influxdb out" node.

    @@ -216,7 +216,7 @@ Double-click the "influxdb out" node to open and configure as follows:
    1. From the "Server" popup menu, choose "Add new influxdb...".
    2. Click the pencil icon to open the server properties panel.
    3. Supply the network path to the host running *InfluxDB*:
    * In a *Docker* environment, this will be the name of the container running **InfluxDB** (eg "influxdb").
    * In a *Docker* environment, this will be the name of the container running *InfluxDB* (eg "influxdb").
    * In a non-*Docker* enviroment where *Node-Red* and *InfluxDB* are running on the same host, this will be the loopback address 127.0.0.1.
    * If *Node-Red* and *InfluxDB* are running on different hosts then this will be the be a static IP address or the fully-qualified domain name of the host running *InfluxDB*.
    4. The name of the *InfluxDB* database. This needs to be the same as you created earlier ("CREATE DATABASE test").
  10. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 4 changed files with 0 additions and 0 deletions.
    Binary file removed change-node.png
    Binary file not shown.
    Binary file removed influxdb-out-node.png
    Binary file not shown.
    Binary file removed mqtt-in-node.png
    Binary file not shown.
    Binary file removed test-flow.png
    Binary file not shown.
  11. @Paraphraser Paraphraser revised this gist Feb 12, 2020. No changes.
  12. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -95,7 +95,7 @@ Back in the *Node-Red* main window, click the "+" button to add a new empty flow

    Drag an "mqtt in" node onto the canvas. Double-click to open and configure as follows:

    ![mqtt-in-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f/raw/6c2ecf4e047a65dae0099d58fd754b93c6dd1914/mqtt-in-node.png)
    ![mqtt-in-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-mqtt-in-node.png)

    1. Select the option to add a new mqtt-broker.
    2. Click the pencil icon to open the server properties panel.
    @@ -211,7 +211,7 @@ Connect the outlet of the "change" node to the inlet of the "influxdb out" node.

    Double-click the "influxdb out" node to open and configure as follows:

    ![influxdb-out-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f/raw/6c2ecf4e047a65dae0099d58fd754b93c6dd1914/influxdb-out-node.png)
    ![influxdb-out-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-influxdb-out-node.png)

    1. From the "Server" popup menu, choose "Add new influxdb...".
    2. Click the pencil icon to open the server properties panel.
    @@ -245,7 +245,7 @@ Click the Deploy button near the top, right of the canvas.

    Add two "debug" nodes to the canvas. Double-click each in turn and set its Output to "complete msg object". Connect the outlet of the "mqtt in" node to the first "debug" node, and the outlet of the "change" node to the second "debug" node. The final result should look something like this:

    ![Node-Red test flow](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f/raw/2473fda0897a094ed7b324a7d537ef39ecadf407/test-flow.png)
    ![Node-Red test flow](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-test-flow.png)

    Click "Deploy" to activate.

  13. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -137,7 +137,7 @@ Connect the outlet of the "mqtt in" node to the inlet of the "change" node.

    Double-click the "change" node to open and configure as follows:

    ![change-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f/raw/bb04382d8aa5d4529f8a286482e4c3a14990068f/change-node.png)
    ![change-node configuration](https://gist.github.com/Paraphraser/1ea84aad35d1d2f57a23c63caee9302f/raw/790ca37f51d4ace9dff149e360e8d6d0fac6d663/mqtt-node-red-influx-change-node.png)

    1. Enter a name for the node. This appears in the schematic and it is good practice to summarise the purpose of the node.
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
  14. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -245,7 +245,7 @@ Click the Deploy button near the top, right of the canvas.

    Add two "debug" nodes to the canvas. Double-click each in turn and set its Output to "complete msg object". Connect the outlet of the "mqtt in" node to the first "debug" node, and the outlet of the "change" node to the second "debug" node. The final result should look something like this:

    ![Node-Red test flow](./test-flow.png)
    ![Node-Red test flow](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f/raw/2473fda0897a094ed7b324a7d537ef39ecadf407/test-flow.png)

    Click "Deploy" to activate.

  15. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -95,7 +95,7 @@ Back in the *Node-Red* main window, click the "+" button to add a new empty flow

    Drag an "mqtt in" node onto the canvas. Double-click to open and configure as follows:

    ![mqtt-in-node configuration](./mqtt-in-node.png)
    ![mqtt-in-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f/raw/6c2ecf4e047a65dae0099d58fd754b93c6dd1914/mqtt-in-node.png)

    1. Select the option to add a new mqtt-broker.
    2. Click the pencil icon to open the server properties panel.
    @@ -211,7 +211,7 @@ Connect the outlet of the "change" node to the inlet of the "influxdb out" node.

    Double-click the "influxdb out" node to open and configure as follows:

    ![influxdb-out-node configuration](./influxdb-out-node.png)
    ![influxdb-out-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f/raw/6c2ecf4e047a65dae0099d58fd754b93c6dd1914/influxdb-out-node.png)

    1. From the "Server" popup menu, choose "Add new influxdb...".
    2. Click the pencil icon to open the server properties panel.
  16. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -137,7 +137,7 @@ Connect the outlet of the "mqtt in" node to the inlet of the "change" node.

    Double-click the "change" node to open and configure as follows:

    ![change-node configuration](change-node.png)
    ![change-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f/raw/bb04382d8aa5d4529f8a286482e4c3a14990068f/change-node.png)

    1. Enter a name for the node. This appears in the schematic and it is good practice to summarise the purpose of the node.
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
  17. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -137,7 +137,7 @@ Connect the outlet of the "mqtt in" node to the inlet of the "change" node.

    Double-click the "change" node to open and configure as follows:

    ![change-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f#file-change-node.png)
    ![change-node configuration](change-node.png)

    1. Enter a name for the node. This appears in the schematic and it is good practice to summarise the purpose of the node.
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
  18. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -137,7 +137,7 @@ Connect the outlet of the "mqtt in" node to the inlet of the "change" node.

    Double-click the "change" node to open and configure as follows:

    ![change-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f#file-change-node-png)
    ![change-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f#file-change-node.png)

    1. Enter a name for the node. This appears in the schematic and it is good practice to summarise the purpose of the node.
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
  19. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -137,7 +137,7 @@ Connect the outlet of the "mqtt in" node to the inlet of the "change" node.

    Double-click the "change" node to open and configure as follows:

    ![change-node configuration](./change-node.png)
    ![change-node configuration](https://gist.github.com/Paraphraser/c9db25d131dd4c09848ffb353b69038f#file-change-node-png)

    1. Enter a name for the node. This appears in the schematic and it is good practice to summarise the purpose of the node.
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
  20. @Paraphraser Paraphraser revised this gist Feb 12, 2020. 5 changed files with 5 additions and 5 deletions.
    10 changes: 5 additions & 5 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -95,7 +95,7 @@ Back in the *Node-Red* main window, click the "+" button to add a new empty flow

    Drag an "mqtt in" node onto the canvas. Double-click to open and configure as follows:

    ![mqtt-in-node configuration](./images/mqtt-in-node.png)
    ![mqtt-in-node configuration](./mqtt-in-node.png)

    1. Select the option to add a new mqtt-broker.
    2. Click the pencil icon to open the server properties panel.
    @@ -137,7 +137,7 @@ Connect the outlet of the "mqtt in" node to the inlet of the "change" node.

    Double-click the "change" node to open and configure as follows:

    ![change-node configuration](./images/change-node.png)
    ![change-node configuration](./change-node.png)

    1. Enter a name for the node. This appears in the schematic and it is good practice to summarise the purpose of the node.
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
    @@ -211,7 +211,7 @@ Connect the outlet of the "change" node to the inlet of the "influxdb out" node.

    Double-click the "influxdb out" node to open and configure as follows:

    ![influxdb-out-node configuration](./images/influxdb-out-node.png)
    ![influxdb-out-node configuration](./influxdb-out-node.png)

    1. From the "Server" popup menu, choose "Add new influxdb...".
    2. Click the pencil icon to open the server properties panel.
    @@ -245,7 +245,7 @@ Click the Deploy button near the top, right of the canvas.

    Add two "debug" nodes to the canvas. Double-click each in turn and set its Output to "complete msg object". Connect the outlet of the "mqtt in" node to the first "debug" node, and the outlet of the "change" node to the second "debug" node. The final result should look something like this:

    ![Node-Red test flow](./images/test-flow.png)
    ![Node-Red test flow](./test-flow.png)

    Click "Deploy" to activate.

    @@ -340,4 +340,4 @@ When you no longer need the test database, you can remove it like this:
    $ influx
    > DROP TABLE test
    > quit
    ```
    ```
    Binary file added change-node.png
    Loading
    Sorry, something went wrong. Reload?
    Sorry, we cannot display this file.
    Sorry, this file is invalid so it cannot be displayed.
    Binary file added influxdb-out-node.png
    Loading
    Sorry, something went wrong. Reload?
    Sorry, we cannot display this file.
    Sorry, this file is invalid so it cannot be displayed.
    Binary file added mqtt-in-node.png
    Loading
    Sorry, something went wrong. Reload?
    Sorry, we cannot display this file.
    Sorry, this file is invalid so it cannot be displayed.
    Binary file added test-flow.png
    Loading
    Sorry, something went wrong. Reload?
    Sorry, we cannot display this file.
    Sorry, this file is invalid so it cannot be displayed.
  21. @Paraphraser Paraphraser created this gist Feb 12, 2020.
    343 changes: 343 additions & 0 deletions MQTT_Node-Red_Influx.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,343 @@
    # Recipe: from MQTT to InfluxDB via Node-Red

    ## Introduction

    Getting data produced by IoT sensors into a database is practically a mandatory step before effective visualisation (eg dashboards).

    This recipe shows you how to get an MQTT payload into an *InfluxDB* database using three *Node-Red* nodes. It makes the following assumptions:

    * A client device of some kind publishing data to a topic via the MQTT protocol;
    * *Mosquitto* (MQTT broker);
    * *Node-Red* subscribing to the topic; and
    * *InfluxDB* running and accessible to *Node-Red*.

    This recipe also show how to map between JSON keys and database field and tag names.


    ## Task goal

    Given this MQTT message structure:

    ```
    topic: /site/topic
    message: {"b": true, "i": 123, "r": 456.78, "s": "hello world", "t": "tagValue"}
    ```

    the task is to configure the *Node-Red* instance to:

    1. Listen for the topic "/site/topic";
    2. Parse the JSON-format message payload; and
    3. Add a row to a measurement (table) named "example" in an *InfluxDB* database named "test".

    This recipe also demonstrates how to use abbreviated keys in a JSON message and map those to more meaningful field names (attributes) in the *InfluxDB* database:

    JSON Key | Influx Field | Influx Tag | Expected Value | Implied type
    :-------:|:------------:|:----------:|:--------------:|:----------------
    b |flag | | true | Boolean
    i |discrete | | 123 | Integer
    r |continuous | | 456.78 | Real
    s |message | | "hello world" | String
    t | | identity | "tagValue" | String

    There is nothing magical about either these JSON keys, or the *InfluxDB* field or tag names. JSON keys do _not_ have to be single characters, and *InfluxDB* names do _not_ have to be long, meaningful or descriptive. It is just that, in general, IoT devices are memory-constrained so short strings are to be preferred over longer strings; while "meaningful and descriptive" are considered *best practice* in a database environment.

    There is also nothing magical about the choice of these implied data types. They just happen to be the ones you are most likely to need to pass between a client device and a database.

    ## Preparation

    ### client device (simulated)

    During testing you will find it useful to have a computer with an MQTT client installed. Explaining the full how-to of this is beyond the scope of this recipe so you should start at [mosquitto.org/download](https://mosquitto.org/download/).

    On Linux, it is not immediately obvious that you need either or both of the following:

    ```
    $ sudo apt install mosquitto-clients
    $ sudo apt install mosquitto
    ```

    The first command installs the MQTT publishing and subscribing command-line clients. You _will_ need this package to get the *mosquitto_pub* command.

    The second command installs the MQTT broker. You will only need this if you do not have another MQTT broker running somewhere (eg in a *Docker* container).

    ### InfluxDB

    If your copy of *InfluxDB* is running inside a *Docker* container, consider adding this *alias* statement to your .profile:

    ```
    $ alias influx='docker exec -it influxdb influx -precision=rfc3339'
    ```

    That alias allows you to connect to the "influx" command line interface (CLI) simply by typing:

    ```
    $ influx
    ```

    The "test" database must be created by hand. If you omit this step you will get an error from *Node-Red*. You can initialise the database like this:

    ```
    $ influx
    > CREATE DATABASE test
    > quit
    ```

    ### Node-Red

    * Launch your browser and connect to your *Node-Red* server.
    * Use the main menu (three horizontal bars "≡" at the top, right of the *Node-Red* window) to open the [Palette Manager](https://nodered.org/docs/user-guide/runtime/adding-nodes), select the "Install" tab, then search for and install [node-red-contrib-influxdb](https://flows.nodered.org/node/node-red-contrib-influxdb). If you prefer to install contributions from the command line, do that.

    ## A three-node flow

    Back in the *Node-Red* main window, click the "+" button to add a new empty flow. The default title will be something like "Flow 1". Double-click the title and rename the flow with something more descriptive, like "Influx Test". The actual name is not important.

    ### 1. "mqtt in" node

    Drag an "mqtt in" node onto the canvas. Double-click to open and configure as follows:

    ![mqtt-in-node configuration](./images/mqtt-in-node.png)

    1. Select the option to add a new mqtt-broker.
    2. Click the pencil icon to open the server properties panel.
    3. Give the server a meaningful name (eg "Docker MQTT").
    4. Supply the network path to the host running *Mosquitto*:
    * In a *Docker* environment, this will be the name of the container running *Mosquitto* (eg "mosquitto").
    * In a non-*Docker* enviroment where *Node-Red* and *Mosquitto* are running on the same host, this will be the loopback address 127.0.0.1.
    * If *Node-Red* and *Mosquitto* are running on different hosts then this will be a static IP address or the fully-qualified domain name of the host running *Mosquitto*.
    5. Click "Add".
    6. Enter a topic (eg "/site/topic").
    7. Set the "Output" popup to "a parsed JSON object").
    8. Enter a name for the node. This appears in the schematic and it is good practice to repeat the topic string.
    9. Click "Done" to complete the node setup.

    All other fields can either be left at their default settings or changed to suit your requirements.

    The purpose of this node is to:

    1. Listen to MQTT messages directed to "/site/topic"; and
    2. Convert the JSON string in the MQTT message body to a JavaScript object representation.

    In other words, given an input of the JSON string specified in the task goal, the output from the node will be:

    ```
    msg.payload = {
    b: true,
    i: 123,
    r: 456.78,
    s: "hello world"
    t: "tagValue"
    }
    ```

    ### 2. "change" node

    Drag a "change" node onto the canvas.

    Connect the outlet of the "mqtt in" node to the inlet of the "change" node.

    Double-click the "change" node to open and configure as follows:

    ![change-node configuration](./images/change-node.png)

    1. Enter a name for the node. This appears in the schematic and it is good practice to summarise the purpose of the node.
    2. A new change node contains a single rule to "Set msg.payload" but where the data type of the "to" field defaults to a string. Change the popup menu to a Java expression.
    3. Click the ellipsis ("…") to open the expression editor.
    4. Copy the expression below and paste it into this window.
    5. Click "Done".
    6. Click "Done" to complete the node setup.

    The expression at step 4 is:

    ```
    [
    {
    "flag": payload.b,
    "discrete": payload.i,
    "continuous": payload.r,
    "message": payload.s
    },{
    "identity": payload.t
    }
    ]
    ```

    The purpose of this node is to provide a cross-walk between the JSON keys ("b", "i", "r", "s" and "t"), and the field and tag names you need in the *InfluxDB* database. The basic pattern is:

    ```
    [
    {
    fieldName : payload.key,
    ...
    },{
    tagName : payload.key,
    ...
    },
    ]
    ```

    If you only want to pass fields, then omit the square brackets and the elements that describe the tag(s), like this:

    ```
    {
    fieldName : payload.key,
    ...
    }
    ```

    Note that it _is_ feasible to omit this "change" node entirely. If you do that the JSON keys in the *Node-Red* "payload" variable will become the field names in the database. But taking that shortcut carries some risks. For example, if an MQTT client starts providing unexpected JSON keys, those can easily pollute your *InfluxDB* database. Using a cross-walk between the expected JSON keys and the field and tag names you want in the database provides an effecive barrier against unexpected keys.

    Given the output from the "mqtt in" node, this Javascript expression in the "change" node will result in:

    ```
    msg.payload = [
    {
    flag: true,
    discrete: 123,
    continuous: 456.78,
    message: "hello world"
    },{
    identity: "tagValue"
    }
    ]
    ```

    ### 3. "influxdb out" node

    Drag an "influxdb out" node onto the canvas.

    > Can't find the "influxdb out" node in the palette? Double-check that you "installed node-red-contrib-influxdb" as described above.
    Connect the outlet of the "change" node to the inlet of the "influxdb out" node.

    Double-click the "influxdb out" node to open and configure as follows:

    ![influxdb-out-node configuration](./images/influxdb-out-node.png)

    1. From the "Server" popup menu, choose "Add new influxdb...".
    2. Click the pencil icon to open the server properties panel.
    3. Supply the network path to the host running *InfluxDB*:
    * In a *Docker* environment, this will be the name of the container running **InfluxDB** (eg "influxdb").
    * In a non-*Docker* enviroment where *Node-Red* and *InfluxDB* are running on the same host, this will be the loopback address 127.0.0.1.
    * If *Node-Red* and *InfluxDB* are running on different hosts then this will be the be a static IP address or the fully-qualified domain name of the host running *InfluxDB*.
    4. The name of the *InfluxDB* database. This needs to be the same as you created earlier ("CREATE DATABASE test").
    5. I recommend leaving this field blank. If you do then the "Server" field in the previous panel will take on the "host:port/database" appearance shown at the end of the dotted line in the figure. You lose that valuable synopsis by supplying a name in this field.
    6. Click "Add".
    7. Supply the name of the measurement you want to write to. This is analogous to a "table" in SQL parlance. The recommended name at this stage is "example".
    8. Enter a name for the node. This appears in the schematic. It is good practice to summarise the purpose of the node.
    9. Click "Done" to complete the node setup.

    All other fields can be left at their default settings or changed to suit your requirements.

    Given the output from the "change" node, the practical effect of this node is:

    ```
    $ influx
    > USE test
    > INSERT example,identity=tagValue flag=true,discrete=123,continuous=456.78,message="hello world"
    > quit
    ```

    ## Saving your work

    Click the Deploy button near the top, right of the canvas.

    ## Testing your work

    Add two "debug" nodes to the canvas. Double-click each in turn and set its Output to "complete msg object". Connect the outlet of the "mqtt in" node to the first "debug" node, and the outlet of the "change" node to the second "debug" node. The final result should look something like this:

    ![Node-Red test flow](./images/test-flow.png)

    Click "Deploy" to activate.

    Select the Debug panel (the controls below "Deploy").

    Copy the following text to the clipboard then paste it into a text editor.

    ```
    mosquitto_pub -h host -t '/site/topic' -m '{"b": true, "i": 123, "r": 456.78, "s": "hello world", "t": "tagValue"}'
    ```

    Edit the "host" field to point to the server running your *Mosquitto* broker. This might be an IP address or a fully-qualified domain name.

    Paste the text into a Terminal window on your client device and press return.

    If all goes well, you will get two debug messages from *Node-Red*. The first is from the "mqtt in" node confirming receipt of the JSON payload:

    ```
    ▿ object
    topic: "/site/topic"
    ▿ payload: object
    b: true
    i: 123
    r: 456.78
    s: "hello world"
    t: "tagValue"
    ```

    and the second is from the "change" node showing the effect of the cross-walk:

    ```
    ▿ object
    topic: "/site/topic"
    ▿ payload: array[2]
    ▿0: object
    flag: true
    discrete: 123
    continuous: 456.78
    message: "hello world"
    ▿1: object
    identity: "tagValue"
    ```

    To confirm that the data made it all the way to the *InfluxDB* database:

    ```
    $ influx
    > USE test
    > show measurements
    name: measurements
    name
    ----
    example
    > show series
    key
    ---
    example,identity=tagValue
    > show tag keys
    name: example
    tagKey
    ------
    identity
    > show field keys
    name: example
    fieldKey fieldType
    -------- ---------
    continuous float
    discrete float
    flag boolean
    message string
    > select * from example
    name: example
    time continuous discrete flag identity message
    ---- ---------- -------- ---- -------- -------
    2020-02-12T03:56:07.844235334Z 456.78 123 true tagValue hello world
    > quit
    ```

    ### Cleaning up

    You can either delete or deactivate the "debug" nodes in the *Node-Red* flow.

    When you no longer need the test database, you can remove it like this:

    ```
    $ influx
    > DROP TABLE test
    > quit
    ```