IoT DataModel for Sensor data - Time-series

[senthilc78 (Senthil C)]

Dear All,
Wishing you all a Very Happy New Year!

Quick one..Recently Ive completed a Data Model for Industrial IoT use case where in ive modeled the Asset Hierarchy as well sensor data . Time series data for every min from around 100 sensors from the Asset (Plant).
My peers are arguing why i need Graph for Time-series. But I firmly believe using Timeseries with Transaction, I can help my customer to find specific faulty part/component among 100 plant at a specific time with low latency as well when i have a History of 20 years sensor data, using my Timeseries Data model i can retrieve my data so quick .

Basically ive created a Node 'SensorValue' which has timestamp and all sensor values (assume 5 sensors values as properties) but ive created Year, Month , Day, Hr and Minute Node as well and attached to SensorValue
(: MINUTE) <-[:REC_MIN]-(sv:SensorValue {Timestamp:"01-01-2019 01:01" , S1Value, S2Value})

(: HOUR) <-[:REC_HR]-(sv) , (: Minute) <-[:REC_MIN]-(sv)
(: Day) <-[:REC_DAY]-(sv)

May have Hr linked with day too but my point is when SensorValue tagged to Time parts, whatever query i feel it wud easy to traverse to specific time slice say
MATCH ( :SensorValue ) -[:REC_DAY]->( :Day {day:17} ) will get sensor data recorded on 17th Day..if i want to narrow down we can filter using YEAR value .

Ive tried for subset of data and it works fine. But would like to know any input/suggestion will it be any better way to do. I was thinking to use fan-out method to partition the nodes to avoid many connection directly connected to 'main Node" if any performance specific question too.

Now AWS announced Time-series database for fast retrieval of time-series ..i feel the principle behind the data-model wud be similar like what im talking here.(since Im crazy in Neo4j)

Please let me know if any one of you tried Data Model for Timeseries and any best practices
and comments or suggestion on my data model.

the problem statement is like "
Get me all the temperature (property) and pressure values (property) on particular day btw 4 pm to 6 pm ( since it was extreme weather condition .) among 5 years of data (record for every 1 min)

Thanks!

Best Regards,
Senthil Chidambaram

[senthilc78 (Senthil C)]

Some Key pointers & updates:

1. Here in my problem statement, ive used aggregated Sensor value (min/max,avg , deviation) per day wise and get it stored in Neo4j and please note , not storing every second 'Telemetry' data to Neo4j since Neo4j is not meant for though it wont restrict you to store but it wud occupy more storage .

2. I did capture only the deviation (low/high ) and got the aggregated value for each type of sensor and connected to get the contextual insights

STEP 1: Build your Asset Hierarchy in Neo4j Graph Model - Instantiate through Cypher batch jobs
STEP 2 : Aggregate the Sensor values (outside Neo4j) for each device per day and only for some critical devices , took hrly aggregated sensor values and link with Day/Hr/Assets nodes but to avoid 'Super dense node' problem
ive managed with multiple relationship types by day wise/week wise/month wise

Take away:
1. In Simple Neo4j is best fit when there is many to many relationship and connecting different Entities/Things (Nodes)
and wants to traverse at 10..25..nth level from Top Nodes to Leaf nodes to get contextual Insights/patterns for better decision and predictions

2. Neo4j is not meant for storing just Telemetry data though you can actually store but other TS or Document db would be better for this purpose

3. Aggregated Metrics/Sensor values, we can take into Neo4j and definitely it would help us to get some known to hidden patterns when we connect different logical entities just not with only Telemetry data

[senthilc78 (Senthil C)]

Some Key pointers & updates:

1. Here in my problem statement, ive used aggregated Sensor value (min/max,avg , deviation) per day wise and get it stored in Neo4j and please note , not storing every second 'Telemetry' data to Neo4j since Neo4j is not meant for though it wont restrict you to store but it wud occupy more storage .

2. I did capture only the deviation (low/high ) and got the aggregated value for each type of sensor and connected to get the contextual insights

STEP 1: Build your Asset Hierarchy in Neo4j Graph Model - Instantiate through Cypher batch jobs
STEP 2 : Aggregate the Sensor values (outside Neo4j) for each device per day and only for some critical devices , took hrly aggregated sensor values and link with Day/Hr/Assets nodes but to avoid 'Super dense node' problem
ive managed with multiple relationship types by day wise/week wise/month wise

Take away:
1. In Simple Neo4j is best fit when there is many to many relationship and connecting different Entities/Things (Nodes)
and wants to traverse at 10..25..nth level from Top Nodes to Leaf nodes to get contextual Insights/patterns for better decision and predictions

2. Neo4j is not meant for storing just Telemetry data though you can actually store but other TS or Document db would be better for this purpose

3. Aggregated Metrics/Sensor values, we can take into Neo4j and definitely it would help us to get some known to hidden patterns when we connect different logical entities just not with only Telemetry data

[senthilc78 (Senthil C)]

Some Key pointers & updates:

1. Here in my problem statement, ive used aggregated Sensor value (min/max,avg , deviation) per day wise and get it stored in Neo4j and please note , not storing every second 'Telemetry' data to Neo4j since Neo4j is not meant for though it wont restrict you to store but it wud occupy more storage .

2. I did capture only the deviation (low/high ) and got the aggregated value for each type of sensor and connected to get the contextual insights

STEP 1: Build your Asset Hierarchy in Neo4j Graph Model - Instantiate through Cypher batch jobs
STEP 2 : Aggregate the Sensor values (outside Neo4j) for each device per day and only for some critical devices , took hrly aggregated sensor values and link with Day/Hr/Assets nodes but to avoid 'Super dense node' problem
ive managed with multiple relationship types by day wise/week wise/month wise

Take away:
1. In Simple Neo4j is best fit when there is many to many relationship and connecting different Entities/Things (Nodes)
and wants to traverse at 10..25..nth level from Top Nodes to Leaf nodes to get contextual Insights/patterns for better decision and predictions

2. Neo4j is not meant for storing just Telemetry data though you can actually store but other TS or Document db would be better for this purpose

3. Aggregated Metrics/Sensor values, we can take into Neo4j and definitely it would help us to get some known to hidden patterns when we connect different logical entities just not with only Telemetry data

[senthilc78 (Senthil C)]

Some Key pointers & updates:

1. Here in my problem statement, ive used aggregated Sensor value (min/max,avg , deviation) per day wise and get it stored in Neo4j and please note , not storing every second 'Telemetry' data to Neo4j since Neo4j is not meant for though it wont restrict you to store but it wud occupy more storage .

2. I did capture only the deviation (low/high ) and got the aggregated value for each type of sensor and connected to get the contextual insights

STEP 1: Build your Asset Hierarchy in Neo4j Graph Model - Instantiate through Cypher batch jobs
STEP 2 : Aggregate the Sensor values (outside Neo4j) for each device per day and only for some critical devices , took hrly aggregated sensor values and link with Day/Hr/Assets nodes but to avoid 'Super dense node' problem
ive managed with multiple relationship types by day wise/week wise/month wise

Take away:
1. In Simple Neo4j is best fit when there is many to many relationship and connecting different Entities/Things (Nodes)
and wants to traverse at 10..25..nth level from Top Nodes to Leaf nodes to get contextual Insights/patterns for better decision and predictions

2. Neo4j is not meant for storing just Telemetry data though you can actually store but other TS or Document db would be better for this purpose

3. Aggregated Metrics/Sensor values, we can take into Neo4j and definitely it would help us to get some known to hidden patterns when we connect different logical entities just not with only Telemetry data

[senthilc78 (Senthil C)]

Some Key pointers & updates:

1. Here in my problem statement, ive used aggregated Sensor value (min/max,avg , deviation) per day wise and get it stored in Neo4j and please note , not storing every second 'Telemetry' data to Neo4j since Neo4j is not meant for though it wont restrict you to store but it wud occupy more storage .

2. I did capture only the deviation (low/high ) and got the aggregated value for each type of sensor and connected to get the contextual insights

STEP 1: Build your Asset Hierarchy in Neo4j Graph Model - Instantiate through Cypher batch jobs
STEP 2 : Aggregate the Sensor values (outside Neo4j) for each device per day and only for some critical devices , took hrly aggregated sensor values and link with Day/Hr/Assets nodes but to avoid 'Super dense node' problem
ive managed with multiple relationship types by day wise/week wise/month wise

Take away:
1. In Simple Neo4j is best fit when there is many to many relationship and connecting different Entities/Things (Nodes)
and wants to traverse at 10..25..nth level from Top Nodes to Leaf nodes to get contextual Insights/patterns for better decision and predictions

2. Neo4j is not meant for storing just Telemetry data though you can actually store but other TS or Document db would be better for this purpose

3. Aggregated Metrics/Sensor values, we can take into Neo4j and definitely it would help us to get some known to hidden patterns when we connect different logical entities just not with only Telemetry data

[senthilc78 (Senthil C)]

Some Key pointers & updates:

1. Here in my problem statement, ive used aggregated Sensor value (min/max,avg , deviation) per day wise and get it stored in Neo4j and please note , not storing every second 'Telemetry' data to Neo4j since Neo4j is not meant for though it wont restrict you to store but it wud occupy more storage .

2. I did capture only the deviation (low/high ) and got the aggregated value for each type of sensor and connected to get the contextual insights

STEP 1: Build your Asset Hierarchy in Neo4j Graph Model - Instantiate through Cypher batch jobs
STEP 2 : Aggregate the Sensor values (outside Neo4j) for each device per day and only for some critical devices , took hrly aggregated sensor values and link with Day/Hr/Assets nodes but to avoid 'Super dense node' problem
ive managed with multiple relationship types by day wise/week wise/month wise

Take away:
1. In Simple Neo4j is best fit when there is many to many relationship and connecting different Entities/Things (Nodes)
and wants to traverse at 10..25..nth level from Top Nodes to Leaf nodes to get contextual Insights/patterns for better decision and predictions

2. Neo4j is not meant for storing just Telemetry data though you can actually store but other TS or Document db would be better for this purpose

3. Aggregated Metrics/Sensor values, we can take into Neo4j and definitely it would help us to get some known to hidden patterns when we connect different logical entities just not with only Telemetry data