Anomaly detection

The goal is to detect anomalies on the basis of given difference of quantity values measured in two different time points. One time point is current time and the second one is selected from data that has been recently measured.

The configuration parameters in the examples below can be changed in the configuration file config.ini that is described in section Configuration file.

CO2

Detection of anomalies is based on difference in carbon dioxide concentration values measured in two various time points.

"""Anomaly detection - CO2.
"""
from os.path import dirname, abspath, join
import sys
sys.path.append(abspath(join(dirname(__file__), '../..', '')))

from dm.ConnectionUtil import ConnectionUtil as cu
from dm.models.ModelsUtil import ModelsUtil
from dm.WundergroundCom import WundergroundCom
from dm.DateTimeUtil import DateTimeUtil


if __name__ == '__main__':
    cu.setup_logging()
    cls = cu.setup_clients()

    actual_time = int(DateTimeUtil.local_time_str_to_utc('2019/02/20 03:00:00').timestamp())
    lat = 49.1649894
    lon = 16.562262499999974

    w = WundergroundCom()
    w.api_key = cu.wunderground_api_key()

    devs = [
        {
            'db_column_name': 'co2_in_ppm',
            'name': '',
            'gateway_id': '1816820318180747',
            'device_id': '0xa900811026800001',
            'module_id': 2,
            'server_name': 'ant-work',
        }
    ]
    notification = ModelsUtil.anomaly_diff(devs, cls, lat, lon, actual_time, w, 2, 7200, 'co2')
    ModelsUtil.json_to_file(notification, 'co2_notification.doc.json', log_notification=True)

It is necessary to set clients that can communicate with a remote server using the function setup_clients. Current time is stored in the variable actual_time. Latitude and longitude that enable to get weather data according to a sensor position are stored in the variable lat and lon. The API key allows to get weather data from server and it is obtained by the function wunderground_api_key. The sensors used to gather data are stored in the variable devs. Anomaly detection is performed by the function anomaly_diff and the result is stored in the variable notification.

An output of the detection is a notification that contains basic information described in the section Notifications and the following information:

  • actual_diff - current difference in carbon dioxide concentration values,

  • anomaly - if anomaly is detected,

  • min_anomaly_diff - minimal difference in carbon dioxide concentration values to detect anomaly,

  • min_anomaly_time - time interval between two time points,

  • type - what quantity is used for prediction.

The example of a notification follows.

{
    "data": {
        "actual_diff": 0.0,
        "anomaly": false,
        "min_anomaly_diff": 2,
        "min_anomaly_time": 120,
        "type": "anomaly_co2"
    },
    "device_id": "0xa900811026800001",
    "event": "env-notification-pre",
    "gateway_id": "1816820318180747",
    "raise": false,
    "raise_catch": false,
    "readable": "2019-02-20 03:00:00",
    "timestamp": 1550628000
}

The example of a command that can be used to detect anomalies on the basis of carbon dioxide concentration difference.

python examples2/0305_anomaly_detection/co2_run.py

Temperature

Detection of anomalies is based on difference in temperature values measured in two various time points.

"""Anomaly detection - temperature.
"""
from os.path import dirname, abspath, join
import sys
sys.path.append(abspath(join(dirname(__file__), '../..', '')))

from dm.ConnectionUtil import ConnectionUtil as cu
from dm.models.ModelsUtil import ModelsUtil
from dm.WundergroundCom import WundergroundCom
from dm.DateTimeUtil import DateTimeUtil


if __name__ == '__main__':
    cu.setup_logging()
    cls = cu.setup_clients()

    actual_time = int(DateTimeUtil.local_time_str_to_utc('2019/02/20 03:00:00').timestamp())
    lat = 49.1649894
    lon = 16.562262499999974

    w = WundergroundCom()
    w.api_key = cu.wunderground_api_key()

    devs = [
        {
            'db_column_name': 'temperature_in_celsius',
            'name': 'BeeeOn sensor',
            'gateway_id': '1816820318180747',
            'device_id': '0xa900811026800001',
            'module_id': 0,
            'server_name': 'ant-work',
        }
    ]
    notification = ModelsUtil.anomaly_diff(devs, cls, lat, lon, actual_time, w, 2, 7200, 'temperature')
    ModelsUtil.json_to_file(notification, 't_notification.doc.json', log_notification=True)

It is necessary to set clients that can communicate with a remote server using the function setup_clients. Current time is stored in the variable actual_time. Latitude and longitude that enable to get weather data according to a sensor position are stored in the variable lat and lon. The API key allows to get weather data from server and it is obtained by the function wunderground_api_key. The sensors used to gather data are stored in the variable devs. Anomaly detection is performed by the function anomaly_diff and the result is stored in the variable notification.

An output of the detection is a notification that contains basic information described in the section Notifications and the following information:

  • actual_diff - current difference in temperature values,

  • anomaly - if anomaly is detected,

  • min_anomaly_diff - minimal difference in temperature values to detect anomaly,

  • min_anomaly_time - time interval between two time points,

  • type - what quantity is used for prediction.

The example of a notification follows.

{
    "data": {
        "actual_diff": 0.22,
        "anomaly": false,
        "min_anomaly_diff": 2,
        "min_anomaly_time": 120,
        "type": "anomaly_temperature"
    },
    "device_id": "0xa900811026800001",
    "event": "env-notification-pre",
    "gateway_id": "1816820318180747",
    "raise": false,
    "raise_catch": false,
    "readable": "2019-02-20 03:00:00",
    "timestamp": 1550628000
}

The example of a command that can be used to detect anomalies on the basis of temperature difference.

python examples2/0305_anomaly_detection/t_run.py

Humidity

Detection of anomalies is based on difference in humidity values measured in two various time points.

"""Anomaly detection - humidity.
"""
from os.path import dirname, abspath, join
import sys
sys.path.append(abspath(join(dirname(__file__), '../..', '')))

from dm.ConnectionUtil import ConnectionUtil as cu
from dm.models.ModelsUtil import ModelsUtil
from dm.WundergroundCom import WundergroundCom
from dm.DateTimeUtil import DateTimeUtil


if __name__ == '__main__':
    cu.setup_logging()
    cls = cu.setup_clients()

    actual_time = int(DateTimeUtil.local_time_str_to_utc('2019/02/20 03:00:00').timestamp())
    lat = 49.1649894
    lon = 16.562262499999974

    w = WundergroundCom()
    w.api_key = cu.wunderground_api_key()

    devs = [
        {
            'db_column_name': 'rh_in_percentage',
            'name': 'BeeeOn sensor',
            'gateway_id': '1816820318180747',
            'device_id': '0xa900811026800001',
            'module_id': 1,
            'server_name': 'ant-work',
        }
    ]
    notification = ModelsUtil.anomaly_diff(devs, cls, lat, lon, actual_time, w, 2, 7200, 'humidity')
    ModelsUtil.json_to_file(notification, 'h_notification.doc.json', log_notification=True)

It is necessary to set clients that can communicate with a remote server using the function setup_clients. Current time is stored in the variable actual_time. Latitude and longitude that enable to get weather data according to a sensor position are stored in the variable lat and lon. The API key allows to get weather data from server and it is obtained by the function wunderground_api_key. The sensors used to gather data are stored in the variable devs. Anomaly detection is performed by the function anomaly_diff and the result is stored in the variable notification.

An output of the detection is a notification that contains basic information described in the section Notifications and the following information:

  • actual_diff - current difference in humidity values,

  • anomaly - if anomaly is detected,

  • min_anomaly_diff - minimal difference in humidity values to detect anomaly,

  • min_anomaly_time - time interval between two time points,

  • type - what quantity is used for prediction.

The example of a notification follows.

{
    "data": {
        "actual_diff": 1.5,
        "anomaly": false,
        "min_anomaly_diff": 2,
        "min_anomaly_time": 120,
        "type": "anomaly_humidity"
    },
    "device_id": "0xa900811026800001",
    "event": "env-notification-pre",
    "gateway_id": "1816820318180747",
    "raise": false,
    "raise_catch": false,
    "readable": "2019-02-20 03:00:00",
    "timestamp": 1550628000
}

The example of a command that can be used to detect anomalies on the basis of humidity difference.

python examples2/0305_anomaly_detection/h_run.py