All Projects → ARMAGEDONgtx → IoT-system-PLC-data-to-InfluxDB

ARMAGEDONgtx / IoT-system-PLC-data-to-InfluxDB

Licence: MIT license
This project aim is to provide free software to fetch data from plcs (Siemens S7-300/400/1200/1500) and store it. Used stack is completly opensource. I used InfluDB as data storage, so application principle is following Big Data paradigm.

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to IoT-system-PLC-data-to-InfluxDB

Logisland
Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.
Stars: ✭ 97 (+273.08%)
Mutual labels:  big-data, influxdb
s7client
Hi level API for node-snap7 to communication with Siemens S7 PLCs
Stars: ✭ 36 (+38.46%)
Mutual labels:  plc, snap7
datalake-etl-pipeline
Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
Stars: ✭ 39 (+50%)
Mutual labels:  big-data
spark-root
Apache Spark Data Source for ROOT File Format
Stars: ✭ 28 (+7.69%)
Mutual labels:  big-data
ghost-on-github-pages
💻⏰ Ghost on Github Pages. Build and deploy Ghost for free in a few minutes.
Stars: ✭ 48 (+84.62%)
Mutual labels:  free
yoda
GitHub extension for agile project management, using the issues subsystem.
Stars: ✭ 86 (+230.77%)
Mutual labels:  free
SmartHome
esp8266 IoT to mqtt bridge. Eastron SDM220, SDM230, SDM630 modbus; AZ7798; SenseAir s8; BME280; HDC1080; Plantower PMS5003, PMS7003, PMSA003
Stars: ✭ 50 (+92.31%)
Mutual labels:  influxdb
gan deeplearning4j
Automatic feature engineering using Generative Adversarial Networks using Deeplearning4j and Apache Spark.
Stars: ✭ 19 (-26.92%)
Mutual labels:  big-data
air-quality
Air quality sensing and monitoring
Stars: ✭ 17 (-34.62%)
Mutual labels:  influxdb
sabnzbd-influxdb-export
Export SABnzbd stats to InfluxDB
Stars: ✭ 16 (-38.46%)
Mutual labels:  influxdb
dxram
A distributed in-memory key-value storage for billions of small objects.
Stars: ✭ 25 (-3.85%)
Mutual labels:  big-data
img2dataset
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
Stars: ✭ 1,173 (+4411.54%)
Mutual labels:  big-data
airavata-django-portal
Mirror of Apache Airavata Django Portal
Stars: ✭ 20 (-23.08%)
Mutual labels:  big-data
coronavirus-dresden
Collects official SARS-CoV-2 infection statistics published by the city of Dresden.
Stars: ✭ 19 (-26.92%)
Mutual labels:  influxdb
lcbo-api
A crawler and API server for Liquor Control Board of Ontario retail data
Stars: ✭ 152 (+484.62%)
Mutual labels:  big-data
eseries-perf-analyzer
This project provides an automated installation and deployment of Grafana, NetApp E-Series Web Services, and supporting software for performance monitoring of NetApp E-Series Storage Systems.
Stars: ✭ 19 (-26.92%)
Mutual labels:  influxdb
Free-Node-Merge
合并网络上的免费节点 | Merge Free ss, ssr, v2ray nodes
Stars: ✭ 154 (+492.31%)
Mutual labels:  free
Real Time Social Media Mining
DevOps pipeline for Real Time Social/Web Mining
Stars: ✭ 22 (-15.38%)
Mutual labels:  big-data
nebula
A distributed, fast open-source graph database featuring horizontal scalability and high availability
Stars: ✭ 8,196 (+31423.08%)
Mutual labels:  big-data
terraform-aws-kinesis-firehose
This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.
Stars: ✭ 25 (-3.85%)
Mutual labels:  big-data

PLC_to_InfluxDB

This project aim is to provide free software to fetch data from plcs (Siemens S7-300/400/1200/1500) and store it. Used stack is completly opensource. I used InfluDB as data storage, which allows to process your data in many interesting ways such as anomaly detecting, forecasting and of course basic data visualization. It also has retention policy, which allows to control your storage space.

For polish users you can read some paper we made about it (just basic explanation and some performance tests) -> https://www.overleaf.com/read/wfqmzmtnbmcb

Structure: struct

Setup steps

  1. Setup machine with InfluxDB database stable realease (1.7.x). You can followe these steps https://devconnected.com/how-to-install-influxdb-on-ubuntu-debian-in-2019/#II_Installing_InfluxDB_20

  2. Install Python3.5+ on the same machine or on some other. Requiremets for Linux and Windows version are in corresponding folders.

Linux

  1. Install Snap7. Here are the steps I followed:

Download and unzip snap7-full-1.4.2
Compile the library
cd snap7-full-1.4.2/build/unix
sudo make -f x86_64_linux.mk Copy the compiled library to lib directories
cd snap7-full-1.4.2/build/x86_64-linux
cp libsnap7.so /usr/lib
cp libsnap7.so /usr/local/lib
Install python-snap7 (pip install python-snap7)

It might be necessery to change common.py from snap7 package if you encouter problems (gijzelaerr/python-snap7#68)

Windows

  1. Install Snap7. I recommend to followe this: https://python-snap7.readthedocs.io/en/latest/installation.html.

Configuration

The data which is fetched from PLCs and send to database is configured via ConfigApp. It generates xml file which is read at the start of program. In configuration you specify (looking from left on below picture, app gives you hints what should you write in field):

  • IP address of PLC from which data will be fetched\
  • RACK slot of the PLC
  • Area of PLC Memory from which we get data:
    -> S7AreaPE - inputs (IW0 etc.)
    -> S7AreaPA - outputs (QW0 etc.)
    -> S7AreaMK - standard memory (MW0 etc.)
    -> S7AreaDB - data blocks (DB10.DBW0 etc.)
    -> S7AreaCT/TM - not tested
  • Data Type of wanted varaible(Bit, Byte, Word ...)
  • Actual address of variable (I0.0 or QW20 etc.)
  • Data alias - alias by which variable will be visible in the system
  • Activate - when activated data will be fetched, if not it will not be

config_app_1 (1)

If you want to add new entry fill requiered fileds. In order to edit actual entries select PLC from drop down menu -> click find aliases -> select data from drop down menu -> clik find data and you will get actual config of selected data. You can edit it or modify. Remember that selected filed before "find data" click is the one that you are editing!

InfluDB endpoint

At the begining of InfluxConnector2.py there is small code fragment, which you need to adjust to your infludb service. Fill it with your confiuration, example below:

config_PATH = '/home/poziadmin/Documents/Python_projects/Linux/config.xml'
influxDB_IP = '10.14.12.83'
influxDB_user = 'admin'
influxDB_pass = 'admin!'

Linux version - asynchronous

Program works as linux deamon with use of systemd (take a look at https://github.com/torfsen/python-systemd-tutorial). It allows to monitor current status and easily start and stop acqusition. For every PLC in configuration is started new process and data is being processed in it. For n PLC is started n Python processes to boost up performance.

To create service we have to make unit file - follow previous tutorial. It will look like this:

[Unit]
Description=Python service to send data from PLC to InfluxDB
PartOf=influxdb.service
After=influxdb.service
[Service]
ExecStart=/path_to_python/Python-3.8.2/python /path_to_app_file/InfluxConnector2.py
Restart=on-failure
[Install]
WantedBy=default.target

Adjust ExecStart variable with your python path and path to the service file.

As the unit file is created we can inspect logs in the future with: sudo journalctl -u PLC2InfluxDB.service

Windows version - synchrnous

In windows version I decided to collect and send data as windows servise. It allows to monitor current status and easily start and stop acqusition. The asynchornous version was dropped due to problems with multiprocessing and service compatibility. So, data is collected one by one and send to database. Choose this version if you do not require fast acqusition.

In order to install service you have to generate exe file with PyInstaller, follow these steps: https://gist.github.com/guillaumevincent/d8d94a0a44a7ec13def7f96bfb713d3f

Visualization capabilities

grafana (1)

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].