cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

closeClose button
November 10, 2016
prev slideNext slide

Edge Intelligence for IoT with Apache MiNiFi

MiNiFI is a subproject of NiFi designed to solve the difficulties of managing and transmitting data feeds to and from the source of origin, often the first/last mile of digital signal, enabling edge intelligence to adjust flow behavior/bi-directional communication.

Since the first mile of data collection (the far edge), is very distributed and likely involves a very large number of end devices (ie. IoT), MiNiFi carries over all the main capabilities of NiFi, with the exception of immediate command and control,  creating a design and deploy paradigm that make uniform management of a vast number of devices more practical. It also means that MiNiFi has a much smaller footprint than NiFi, with a range less than 40MB, depending which option is selected – MiNiFi with the Java Agent, or a Native C++ agent.

edge-intelligence-for-iot-with-apache-minifi

Highlights of Apache MiNiFi

  1. Configuration is driven through a declarative configuration file that allows for popular infrastructure automation tools to facilitate the process and management of larger numbers of instances.
  2. MiNiFi agents make use of the Site to Site Protocol to aid in the transmission of data from where it is collected to a core NiFi installation for handling distribution and delivery to consumers throughout your organization’s enterprise.
  3. While the focus is on smaller distributions, MiNiFi Java can make use of existing NiFi NAR bundles and extensions.  Users can get the same functionalities in data source cases where design and deploy fits the mechanics of their dataflow needs
  4. Bi-directional communication enables dynamic updates of processing flow to facilitate changes to processing through design and deploy as well as distilling generated insights into models that can be applied before data is transmitted back

Some example use cases of Apache MiNiFi

MiNiFi is ideal for situations where there are a large number of distributed, connected devices. For example first mile data collection from routers, security cameras, cable modems, ATMs, security appliances, point of sale systems, weather detection systems, fleets of trucks/trains/planes/ships, thermostats, utility or power meters are all good uses cases for MiNiFi due the distributed nature of their physical locations.  MiNIFi then enables multiple applications. For example, by deploying MiNiFi on point of sale terminals, retailers can gather data from 100’s or 1000’s or more terminals and then obtain:

  • Real-time pricing data
  • Real-time inventory updates
  • Create real-time offers
  • Access traceable logs of transactions to understand usage, behavior, buying patterns
  • Capture a log of user-interactions for trouble-shooting of POS devices
  • Support predictive help desk service to improve reduce downtime and improve customer satisfaction

Another example is the connected car. By deploying MiNiFi on automobiles one can:

A third use case is within communications companies  – for example to capture real-time user transactions with television programming. Knowing what content is being viewed, when users change channels or switch alternatives provides the ability to understand user behaviour, assess the quality of video being delivered (ie. is it buffering or stalling), create viewing recommendations on real-time and dynamically serve relevant ads.

apache-minifi-use-cases-hortonworks-hdf

 

Slideshare with more information:

The number of IoT type applications is broad and diverse, but all depend on connectivity, connectivity that is assumed to be perfect but in reality is not. MiNiFi (and NiFi) address this problem, and all the corresponding requirements of managing and transmitting data feeds to and from the first/last digital mile, thus enabling edge intelligence in real-time to create brand new business opportunities from previously inaccessible data.

For more information:

Should you have any more questions, anytime, we encourage you to check out the Data Ingestion & Streaming track of Hortonworks Community Connection where an entire community of folks are monitoring and responding to questions. You may also want to check out our HDF home pageHDF support and HDF operations training pages.

Q&A

1. With NiFi and MiNiFi, do we get dynamic heap memory allocation for JVM?

We would like to better understand your question to be able to answer as this could mean a couple different scenarios. If you like – please post your questions to Hortonworks Community Connection.

2. How can I install MiNiFi in my Nest?

We’ve not done any specific work there yet but would be happy to discuss it further.

3. Are there any limitations on number of MiNiFi agents?

No, there are no limits. The number of agents is based on your needs, your allocation of resources. From a use case perspective, you can certainly deploy hundreds of thousands of MiNiFi agents on your edge devices. Think of moving vehicles, routers/switches, PoS systems, smart refrigerators, etc. Typically, you will probably face massive challenges in terms of resource limitations, network bandwidth limitation on the edge, which is exactly why a prioritization mechanism is built into the MiNiFi framework to handle the first mile data collection.

4. Can MiNiFi handle multiple devices/flows at the edge?

Yes, MiNiFi can support multiple flows at the edge. The objective is a distributed agent model with enterprise readiness. Moving forward, a centralized command and control console can further enhance user experience. Essentially you can push a flow designed in NIFI to multiple MiNiFi agents via the UI, and be able to version the flows running on the edge. For more details, please refer to the feature proposal available in Apache: https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control

Regarding  multiple devices, each with their MiNiFi flow can be combined into a single flow.  One MiNiFi could also feed data to another MiNiFi using various protocols (ie a push and receive processor respectively, ex: HTTPS) but currently S2S input is not supported by MiNiFi.

5. Can you create a NiFi flow via the NiFI UI and then export to the MiNiFi YAML format?

Yes, this can be accomplished by using the MiNiFi toolkit to convert a NiFi template into a MiNiFi config file. (see Getting started with MiNiFi). Also see question above for more details.

6. What if you want to use MiNiFi to tail more than one system file?

Yes, you can tail more than one system files, by having two disconnected graphs, or having two different tail file processors that are configured to tail two different systems. Refer to the webinar at 44 min.

7. Does MiNiFi have cert management?

NiFi and MiNiFi use PKI certification for site to site communication as well as the configuration change notifications (HTTPS with certs). However, there is currently no command and control based certification management.

8. Can you get an alert if an agent stops working?

This capability is tied to the command and control capabilities of NiFi, which is planned for the next phase of MiNiFi. (more info in Apache here: https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control)

As a temporary alternative you could leverage the MiNiFi provenance capabilities – see the 45 minute point of the live webinar for reference.

9. Is MiNiFi valid only for sensors communicating using IT protocols? What about legacy sensors using something like I2C (eye-square-see) ?

Yes. It is important to note that both NiiFi and MiNiFi are frameworks designed and built for extensibility and to support custom components. Therefore MiNiFi can be valid for legacy sensors too. A processor just needs to be written implementing their respective API.   The developer guide for the NiFi and MiNiFi java framework is here: http://nifi.apache.org/developer-guide.html. The C++ information will be made available as that core framework is finalized.

10. What are the other Hadoop ecosystem components or any other components required for a working NiFi or MiNiFi setup ?

No other Hadoop ecosystem components are required for a working NiFi or MiNiFi setup. It will also be easier to utilize a NiFi instance to design a flow/template and convert to MiNiFi config.  That being said, you can easily access Hadoop and related technologies as part of Apache NiFi and Apache MiNiFi. Also see the response to this question here: “Using HDF alone, can we do analytics on data in motion. Or does it have to get into hadoop for doing analytics?

Leave a Reply

Your email address will not be published. Required fields are marked *