Open Contrail (Architecture and working) Analytics node

Open Contrail

Nodes

Analytics nodes collect, store, correlate, and analyze information from network elements, virtual or physical. This information includes statistics, logs, events, and errors.

Configuration Node:

Figure 6 shows the internal structure of a configuration node. The configuration node communicates with the orchestration system via a REST interface, with other configuration nodes via a distributed synchronization mechanism, and with control nodes via IF-MAP.

Configuration nodes also provide a discovery service that the clients can use to locate the service providers (that is, other nodes providing a particular service). For example, when the vRouter agent in a compute node connects to a control node1, it uses service discovery to discover the IP address of the control node. The clients use local configuration, DHCP or DNS to locate the service discovery server.

Configuration nodes contain the following components:

A REST API server provides the northbound interface to an orchestration system or other application. This interface is used to install configuration state using the high-level data model.

A Redis [redis] message bus facilitates communications among internal components.

A Cassandra [cassandra] database provides persistent storage of configuration. Cassandra is a fault-tolerant and horizontally scalable database.

A Schema transformer detects changes in the high-level data model over the Redis message bus and transforms (or compiles) these changes in the high-level data model into corresponding changes in the low-level data model.

ZooKeeper [zookeeper] (not shown in diagram) is used for allocation of unique object identifiers and to implement transactions.

Analytics Node:

Figure 7 shows the internal structure of an analytics node. An analytics node communicates with applications using a northbound REST API (northbound interface is an interface that allows a particular component of a network communicate with a higher-level component), with other analytics nodes using a distributed synchronization mechanism, and with components in control and configuration nodes using an XML-based protocol called Sandesh designed specifically for handling high volumes of data.

Sandesh is a southbound interface protocol that is used by the Open Contrail Analytics engine to gather information from the Open Contrail virtual routers and other software modules like the control-node, and the configuration manager, which run as part of the controller.

The analytics nodes contain the following components:

A collector exchanges Sandesh messages with components in control nodes and configuration nodes to collect analytics information.

A NoSQL database stores this information.

A rules engine automatically collects operational state when specific events occur.

A REST API server provides a northbound interface for querying the analytics database and for retrieving operational state.

A query engine executes the queries received over the northbound REST API. This engine provides the capability for flexible access to potentially large amounts of analytics data.

Sandesh carries two kinds of messages—asynchronous messages received by analytics nodes for the purpose of reporting logs, events, and traces; and synchronous messages whereby an analytics node can send requests and receive responses to collect specific operational state.

All information gathered by the collector is persistently stored in the NoSQL database. No filtering of messages is done by the information source. The analytics nodes provide a northbound REST API to allow client applications to submit queries.

Explanation:

An analytics node communicates with applications using a northbound REST API, with other analytics nodes using a distributed synchronization mechanism, and with components in control and configuration nodes using an XML-based protocol known as Sandesh designed specifically for handling high volumes of data.

The collector collects messages sent by the sandesh to collect the analytics data and this data is stored in the NoSQL database. To get all the information from the sandesh these messages have not passed through filters. When specific events occur rules engine automatically collects operational state. A REST API server provides a northbound interface for querying the analytics database and retrieving the operational state from the database information. A query engine execute the queries received over the northbound REST API. This query engine provides the capability for flexible access to potentially large amounts of analytics data. The data is analyzed by the analytics nodes one by one and the process goes on.

REST API is provided by analytics nodes to allow client applications to submit queries. Analytics nodes provide scatter-gather logic called “aggregation.” A single GET request (and a single corresponding CLI command in the client application) can be mapped to multiple request messages whose results are combined. The query engine is implemented as a simple map-reduce engine. The vast majority of Contrail queries are time series (a series of data points indexed)

Redis Structure

Redsi is use as a message bus in contrail system. It can use for many purposes and there are many key-value stores. Contrail populate 4 different key-value stores.

There names are

• DB1
• DB3
• DB4
• DB7

DB1

This key value store use to store querries coming from API servers. Then querry engine read querry from DB1 and perform it on Database. After executing querry Querry engine store data in DB1.

DB3:

This key-value store use to store flow records

DB4:

This key-value store use to store UVES in xml form that we see on port 8081

DB7:

This key-value store is same as what we see on port 8081

Note: The data store in all the key-value store is distributed using keys and values concept. So, if you want to retrive data you need key for that data. If you are a subscriber you received key where the change occurs and then you perform query on redis to get data.

Cassandra Structure

There are many databases in Cassandra but we focus on Analytics Data base which is name as ContrailAnalyticsCql

ContrailAnalyticsCql Database:

Now, these are the tables in contrail analytics database

flowrecordtable flowtablevrouterver2 statstablebydbltagv3

objectvaluetable messagetablemessagetype messagetablekeyword

messagetabletimestamp messagetablecategory statstablebyu64strtagv3

statstablebyu64tagv3 messagetablesource statstablebystrstrtagv3

systemobjecttable flowtableprotdpver2 statstablebystru64tagv3

objecttable statstablebyu64u64tagv3 flowtabledvndipver2

messagetable statstablebystrtagv3 flowtableprotspver2

flowtablesvnsipver2 messagetablemoduleid

Now we list all tables with their attributes

flowrecordtable

• action
• bytes
• data_sample
• destip
• destvn
• direction_ing
• dport
• drop_reason
• input_interface
• key
• max_interarrival
• mean_interarrival
• min_interarrival
• mpls_label
• nw_ace_uuid
• other_vrouter_ip
• output_interface
• packets
• protocol
• reverse_uuid
• setup_time
• sg_rule_uuid
• sourceip
• sourcevn
• sport
• stddev_interarrival
• tcp_flags
• teardown_time
• tos
• underlay_proto
• underlay_source_port
• vm
• vmi_uuid
• vrouter
• vrouter_ip

flowtablevrouterver2

• column1
• column2
• column3
• key
• key2
• key3
• value

statstablebydbltagv3

• column1
• column2
• column3
• key
• key2
• key3
• key4
• key5
• value

objectvaluetable

• column1
• key
• key2
• value

Messagetablemessagetype

• column1
• column2
• column3
• key
• key2

Messagetablekeyword

• column1
• column2
• column3
• key2

messagetabletimestamp

• column1
• column2
• key
• key2

messagetablecategory

• column1
• column2
• column3
• key
• key2

statstablebyu64strtagv3

• column1
• column2
• column3
• column4
• key
• key2
• key3
• key4
• key5
• key6
• value

statstablebyu64tagv3

• column1
• column2
• column3
• key
• key2
• key3
• key4
• key5
• value

messagetablesource

• column1
• column2
• column3
• key
• key2

statstablebystrstrtagv3

• column1
• column2
• column3
• column4
• key
• key2
• key3
• key4
• key5
• key6
• value

systemobjecttable

• SystemObjectConfigAuditTtl
• SystemObjectFlowDataTtl
• SystemObjectFlowStartTime
• SystemObjectGlobalDataTtl
• SystemObjectMsgStartTime
• SystemObjectStartTime
• SystemObjectStatStartTime
• SystemObjectStatsDataTtl
• key

flowtableprotdpver2:

• flowtableprotdpver2
• column1
• column2
• column3
• column4
• key
• key2
• key3
• value

statstablebystru64tagv3

• column1
• column2
• column3
• column4
• key
• key2
• key3
• key4
• key5
• key6
• value

objecttable

• column1
• column2
• key
• key2
• key3
• value

statstablebyu64u64tagv3

• column1
• column2
• column3
• column4
• key
• key2
• key3
• key4
• key5
• key6
• value

flowtabledvndipver2

• column1
• column2
• column3
• column4
• key
• key2
• key3
• value

messagetable

• Category
• Context
• IPAddress
• InstanceId
• Level
• MessageTS
• Messagetype
• ModuleId
• Namespace
• NodeType
• Pid
• SequenceNum
• Source
• Type
• VersionSig
• Xmlmessage
• key

statstablebystrtagv3

• column1
• column2
• column3
• key
• key2
• key3
• key4
• key5
• value

flowtableprotspver2

• column1
• column2
• column3
• column4
• key
• key2
• key3
• value

flowtablesvnsipver2

• column1
• column2
• column3
• column4
• key
• key2
• key3
• value

messagetablemoduleid

• column1
• column2
• column3
• key
• key2