Publication detail

Low Overhead Distributed IP Flow Records Collection and Analysis

WRONA, J. ŽÁDNÍK, M.

Original Title

Low Overhead Distributed IP Flow Records Collection and Analysis

Type

conference paper

Language

English

Original Abstract

Collection and analysis of IP flow records belong to a class of data-intensive tasks, the class for which big data analytics systems should be effective. Several Hadoop-based solutions for network traffic processing exist but are generally suitable only for truly big data, otherwise the disadvantages of Hadoop dominate. In this work, we present a distributed platform for IP flow records collection and analysis together with a reference implementation. It focuses on smaller clusters, has low overhead, allows interactive work, and exploits the prospects of distributed systems like high throughput and scalability. Experiments show low query latency and linear scalability with respect to the growth of both amount of work and computer cluster. Extensions for data mining and machine learning are easy to include and are already work in progress. Moreover, the whole software stack is open-source.

Keywords

NetFlow, IPFIX, IP flow collector, distributed system, parallel computing, Hadoop, big data

Authors

WRONA, J.; ŽÁDNÍK, M.

Released

8. 4. 2019

Location

Washington DC

ISBN

978-3-903176-15-7

Book

2019 IFIP/IEEE International Symposium on Integrated Network Management

Pages from

557

Pages to

562

Pages count

6

URL

https://ieeexplore.ieee.org/document/8717873

BibTex

@inproceedings{BUT161793,
  author="Jan {Wrona} and Martin {Žádník}",
  title="Low Overhead Distributed IP Flow Records Collection and Analysis",
  booktitle="2019 IFIP/IEEE International Symposium on Integrated Network Management",
  year="2019",
  pages="557--562",
  address="Washington DC",
  isbn="978-3-903176-15-7",
  url="https://ieeexplore.ieee.org/document/8717873"
}