Tuesday, 24 October 2017

Open Contrail (Architecture and working) Analytics node

21:46 Posted by SRE Hacks , , , , , , , , , , , No comments
Open Contrail
Nodes
Analytics nodes collect, store, correlate, and analyze information from network elements, virtual or physical. This information includes statistics, logs, events, and errors.
Configuration Node:
Figure 6 shows the internal structure of a configuration node. The configuration node communicates with the orchestration system via a REST interface, with other configuration nodes via a distributed synchronization mechanism, and with control nodes via IF-MAP.
Configuration nodes also provide a discovery service that the clients can use to locate the service providers (that is, other nodes providing a particular service). For example, when the vRouter agent in a compute node connects to a control node1, it uses service discovery to discover the IP address of the control node. The clients use local configuration, DHCP or DNS to locate the service discovery server.




Configuration nodes contain the following components:
A REST API server provides the northbound interface to an orchestration system or other application. This interface is used to install configuration state using the high-level data model.
A Redis [redis] message bus facilitates communications among internal components.
A Cassandra [cassandra] database provides persistent storage of configuration. Cassandra is a fault-tolerant and horizontally scalable database.
A Schema transformer detects changes in the high-level data model over the Redis message bus and transforms (or compiles) these changes in the high-level data model into corresponding changes in the low-level data model.
ZooKeeper [zookeeper] (not shown in diagram) is used for allocation of unique object identifiers and to implement transactions.
Analytics Node:
Figure 7 shows the internal structure of an analytics node. An analytics node communicates with applications using a northbound REST API (northbound interface is an interface that allows a particular component of a network communicate with a higher-level component), with other analytics nodes using a distributed synchronization mechanism, and with components in control and configuration nodes using an XML-based protocol called Sandesh designed specifically for handling high volumes of data.
Sandesh is a southbound interface protocol that is used by the Open Contrail Analytics engine to gather information from the Open Contrail virtual routers and other software modules like the control-node, and the configuration manager, which run as part of the controller.
The analytics nodes contain the following components:
A collector exchanges Sandesh messages with components in control nodes and configuration nodes to collect analytics information.
A NoSQL database stores this information.
A rules engine automatically collects operational state when specific events occur.
A REST API server provides a northbound interface for querying the analytics database and for retrieving operational state.
A query engine executes the queries received over the northbound REST API. This engine provides the capability for flexible access to potentially large amounts of analytics data.
Sandesh carries two kinds of messages—asynchronous messages received by analytics nodes for the purpose of reporting logs, events, and traces; and synchronous messages whereby an analytics node can send requests and receive responses to collect specific operational state.
All information gathered by the collector is persistently stored in the NoSQL database. No filtering of messages is done by the information source. The analytics nodes provide a northbound REST API to allow client applications to submit queries.
Explanation:
An analytics node communicates with applications using a northbound REST API, with other analytics nodes using a distributed synchronization mechanism, and with components in control and configuration nodes using an XML-based protocol known as Sandesh designed specifically for handling high volumes of data.
The collector collects messages sent by the sandesh to collect the analytics data and this data is stored in the NoSQL database. To get all the information from the sandesh these messages have not passed through filters. When specific events occur rules engine automatically collects operational state. A REST API server provides a northbound interface for querying the analytics database and retrieving the operational state from the database information. A query engine execute the queries received over the northbound REST API. This query engine provides the capability for flexible access to potentially large amounts of analytics data. The data is analyzed by the analytics nodes one by one and the process goes on.
REST API is provided by analytics nodes to allow client applications to submit queries. Analytics nodes provide scatter-gather logic called “aggregation.” A single GET request (and a single corresponding CLI command in the client application) can be mapped to multiple request messages whose results are combined. The query engine is implemented as a simple map-reduce engine. The vast majority of Contrail queries are time series (a series of data points indexed)

Redis Structure



Redsi is use as a message bus in contrail system. It can use for many purposes and there are many key-value stores. Contrail populate 4 different key-value stores.
There names are
  • DB1
  • DB3
  • DB4
  • DB7
DB1
This key value store use to store querries coming from API servers. Then querry engine read querry from DB1 and perform it on Database. After executing querry Querry engine store data in DB1.


DB3:
This key-value store use to store flow records


DB4:
This key-value store use to store UVES in xml form that we see on port 8081
DB7:
This key-value store is same as what we see on port 8081


Note: The data store in all the key-value store is distributed using keys and values concept. So, if you want to retrive data you need key for that data. If you are a subscriber you received key where the change occurs and then you perform query on redis to get data.

Cassandra Structure

There are many databases in Cassandra but we focus on Analytics Data base which is name as ContrailAnalyticsCql
ContrailAnalyticsCql Database:
Now, these are the tables in contrail analytics database


flowrecordtable flowtablevrouterver2 statstablebydbltagv3
objectvaluetable messagetablemessagetype messagetablekeyword
messagetabletimestamp messagetablecategory statstablebyu64strtagv3
statstablebyu64tagv3 messagetablesource statstablebystrstrtagv3
systemobjecttable flowtableprotdpver2 statstablebystru64tagv3
objecttable statstablebyu64u64tagv3 flowtabledvndipver2
messagetable statstablebystrtagv3 flowtableprotspver2
flowtablesvnsipver2 messagetablemoduleid
Now we list all tables with their attributes
flowrecordtable
  • action
  • bytes
  • data_sample
  • destip
  • destvn
  • direction_ing
  • dport
  • drop_reason
  • input_interface
  • key
  • max_interarrival
  • mean_interarrival
  • min_interarrival
  • mpls_label
  • nw_ace_uuid
  • other_vrouter_ip
  • output_interface
  • packets
  • protocol
  • reverse_uuid
  • setup_time
  • sg_rule_uuid
  • sourceip
  • sourcevn
  • sport
  • stddev_interarrival
  • tcp_flags
  • teardown_time
  • tos
  • underlay_proto
  • underlay_source_port
  • vm
  • vmi_uuid
  • vrouter
  • vrouter_ip


flowtablevrouterver2
  • column1
  • column2
  • column3
  • key
  • key2
  • key3
  • value
statstablebydbltagv3
  • column1
  • column2
  • column3
  • key
  • key2
  • key3
  • key4
  • key5
  • value
objectvaluetable
  • column1
  • key
  • key2
  • value
Messagetablemessagetype
  • column1
  • column2
  • column3
  • key
  • key2
Messagetablekeyword
  • column1
  • column2
  • column3
  • key2
messagetabletimestamp
  • column1
  • column2
  • key
  • key2
messagetablecategory
  • column1
  • column2
  • column3
  • key
  • key2
statstablebyu64strtagv3
  • column1
  • column2
  • column3
  • column4
  • key
  • key2
  • key3
  • key4
  • key5
  • key6
  • value
statstablebyu64tagv3
  • column1
  • column2
  • column3
  • key
  • key2
  • key3
  • key4
  • key5
  • value
messagetablesource
  • column1
  • column2
  • column3
  • key
  • key2
statstablebystrstrtagv3
  • column1
  • column2
  • column3
  • column4
  • key
  • key2
  • key3
  • key4
  • key5
  • key6
  • value
systemobjecttable
  • SystemObjectConfigAuditTtl
  • SystemObjectFlowDataTtl
  • SystemObjectFlowStartTime
  • SystemObjectGlobalDataTtl
  • SystemObjectMsgStartTime
  • SystemObjectStartTime
  • SystemObjectStatStartTime
  • SystemObjectStatsDataTtl
  • key
flowtableprotdpver2:
  • flowtableprotdpver2
  • column1
  • column2
  • column3
  • column4
  • key
  • key2
  • key3
  • value
statstablebystru64tagv3
  • column1
  • column2
  • column3
  • column4
  • key
  • key2
  • key3
  • key4
  • key5
  • key6
  • value
objecttable
  • column1
  • column2
  • key
  • key2
  • key3
  • value
statstablebyu64u64tagv3
  • column1
  • column2
  • column3
  • column4
  • key
  • key2
  • key3
  • key4
  • key5
  • key6
  • value
flowtabledvndipver2
  • column1
  • column2
  • column3
  • column4
  • key
  • key2
  • key3
  • value


messagetable
  • Category
  • Context
  • IPAddress
  • InstanceId
  • Level
  • MessageTS
  • Messagetype
  • ModuleId
  • Namespace
  • NodeType
  • Pid
  • SequenceNum
  • Source
  • Type
  • VersionSig
  • Xmlmessage
  • key
statstablebystrtagv3
  • column1
  • column2
  • column3
  • key
  • key2
  • key3
  • key4
  • key5
  • value
flowtableprotspver2
  • column1
  • column2
  • column3
  • column4
  • key
  • key2
  • key3
  • value
flowtablesvnsipver2
  • column1
  • column2
  • column3
  • column4
  • key
  • key2
  • key3
  • value
messagetablemoduleid

  • column1
  • column2
  • column3
  • key
  • key2