Friday, 21 April 2017

Bash scripting

Introduction

This page is mostly foundation information. It's kinda boring but essential stuff that will help you to appreciate why and how certian things behave the way they do once we start playing about with the fun stuff (which I promise we'll do in the next section). Taking the time to read and understand the material in this section will make the other sections easier to digest so persevere and it'll be well worth your time.

So what are they exactly?

Think of a script for a play, or a movie, or a TV show. The script tells the actors what they should say and do. A script for a computer tells the computer what it should do or say. In the context of Bash scripts we are telling the Bash shell what it should do.
A Bash script is a plain text file which contains a series of commands. These commands are a mixture of commands we would normally type ouselves on the command line (such as ls or cp for example) and commands we could type on the command line but generally wouldn't (you'll discover these over the next few pages). An important point to remember though is:
Anything you can run normally on the command line can be put into a script and it will do exactly the same thing. Similarly, anything you can put into a script can also be run normally on the command line and it will do exactly the same thing.
You don't need to change anything. Just type the commands as you would normally and they will behave as they would normally. It's just that instead of typing them at the command line we are now entering them into a plain text file. In this sense, if you know how to do stuff at the command line then you already know a fair bit in terms of Bash scripting.
It is convention to give files that are Bash scripts an extension of .sh (myscript.sh for example). As you would be aware (and if you're not maybe you should consider reviewing our Linux Tutorial), Linux is an extensionless system so a script doesn't necessarily have to have this characteristic in order to work.

How do they work?

This is just a little bit of background knowledge. It's not necessary to understand this in order to write scripts but it can be useful to know once you start getting into more complex scripts (and scripts that call and rely on other scripts once you start getting really fancy).
In the realm of Linux (and computers in general) we have the concept of programs and processes. A program is a blob of binary data consisting of a series of instructions for the CPU and possibly other resources (images, sound files and such) organised into a package and typically stored on your hard disk. When we say we are running a program we are not really running the program but a copy of it which is called a process. What we do is copy those instructions and resources from the hard disk into working memory (or RAM). We also allocate a bit of space in RAM for the process to store variables (to hold temporary working data) and a few flags to allow the operating system (OS) to manage and track the process during it's execution.
Essentially a process is a running instance of a program.
There could be several processes representing the same program running in memory at the same time. For example I could have two terminals open and be running the command cp in both of them. In this case there would be two cp processes currently existing on the system. Once they are finished running the system then destroys them and there are no longer any processes representing the program cp.
When we are at the terminal we have a Bash process running in order to give us the Bash shell. If we start a script running it doesn't actually run in that process but instead starts a new process to run inside. We'll demonstrate this in the next section on variables and it's implications should become clearer. For the most part you don't need to worry too much about this phenomenon however.

How do we run them?

Running a Bash script is fairly easy. Another term you may come across is executing the script (which means the same thing). Before we can execute a script it must have the execute permission set (for safety reasons this persmission is generally not set by default). If you forget to grant this permission before running the script you'll just get an error message telling you as such and no harm will be done.
  1. ./myscript.sh
  2. bash: ./myscript.sh: Permission denied
  3. ls -l myscript.sh
  4. -rw-r--r-- 18 ryan users 4096 Feb 17 09:12 myscript.sh
  5. chmod 755 myscript.sh
  6. ls -l myscript.sh
  7. -rwxr-xr-x 18 ryan users 4096 Feb 17 09:12 myscript.sh
  8. ./myscript.sh
  9. Hello World!
The shorthand 755 is often used for scripts as it allows you the owner to write or modify the script and for everyone to execute the script.
Here are the contents of myscript.sh

myscript.sh

  1. #!/bin/bash
  2. # A sample Bash script, by Ryan
  3. echo Hello World!
Let's break it down:

  • Line 1 - Is what's referred to as the shebang. See below for what this is.
  • Line 2 - This is a comment. Anything after # is not executed. It is for our reference only.
  • Line 4 - Is the command echo which will print a message to the screen. You can type this command yourself on the command line and it will behave exactly the same.
  • The syntax highlighting is there only to make it easier to read and is not something you need to do in your own files (remember they are just plain text files).

Why the ./

You've possibly noticed that when we run a normal command (such as ls) we just type its name but when running the script above I put a ./ in front of it. When you just type a name on the command line Bash tries to find it in a series of directories stored in a variable called $PATH. We can see the current value of this variable using the command echo (you'll learn more about variables in the next section).
  1. echo $PATH
  2. /home/ryan/bin:/usr/local/bin:/usr/bin:/bin
The directories are separated by " : "
Bash only looks in those specific directories and doesn't consider sub directories or your current directory. It will look through those directories in order and execute the first instance of the program or script that it finds.
The $PATH variable is an individual user variable so each user on a system may set it to suit themselves.
This is done for a few different reasons.
  • It allows us to have several different versions of a program installed. We can control which one gets executed based on where it sits in our $PATH.
  • It allows for convenience. As you saw above, the first directory for myself is a bin directory in my home directory. This allows me to put my own scripts and programs there and then I can use them no matter where I am in the system by just typing their name. I could even create a script with the same name as a program (to act as a wrapper) if I wanted slightly different behaviour.
  • It increases safety - For example a malicious user could create a script called ls which actually deletes everything in your home directory. You wouldn't want to inadvertantly run that script. But as long as it's not in your $PATH that won't happen.
If a program or script is not in one of the directories in your $PATH then you can run it by telling Bash where it should look to find it. You do so by including either an absolute or relative path in front of the program or script name. You'll remember that dot ( . ) is actually a reference to your current directory. Assuming this script is in my home directory I could also have run it by using an absolute path.
  1. /home/ryan/myscript.sh
  2. Hello World!
Variables

In this example we declare simple bash variable and print it on the screen ( stdout ) with echo command.

#!/bin/bash
 STRING="HELLO WORLD!!!"
 echo $STRING 

Bash string Variables in bash script

Your backup script and variables:

#!/bin/bash
 OF=myhome_directory_$(date +%Y%m%d).tar.gz
 tar -czf $OF /home/linuxconfig 

Bash backup Script with bash Variables

Global vs. Local variables

#!/bin/bash
#Define bash global variable
#This variable is global and can be used anywhere in this bash script
VAR="global variable"
function bash {
#Define bash local variable
#This variable is local to bash function only
local VAR="local variable"
echo $VAR
}
echo $VAR
bash
# Note the bash global variable did not change
# "local" is bash reserved word
echo $VAR

Global vs. Local Bash variables in bash script
Passing arguments to the bash script

#!/bin/bash
# use predefined variables to access passed arguments
#echo arguments to the shell
echo $1 $2 $3 ' -> echo $1 $2 $3'

# We can also store arguments from bash command line in special array
args=("$@")
#echo arguments to the shell
echo ${args[0]} ${args[1]} ${args[2]} ' -> args=("$@"); echo ${args[0]} ${args[1]} ${args[2]}'

#use $@ to print out all arguments at once
echo $@ ' -> echo $@'

# use $# variable to print out
# number of arguments passed to the bash script
echo Number of arguments passed: $# ' -> echo Number of arguments passed: $#' 

/arguments.sh Bash Scripting Tutorial 

Passing arguments to the bash script
Executing shell commands with bash

#!/bin/bash
# use backticks " ` ` " to execute shell command
echo `uname -o`
# executing bash command without backticks
echo uname -o 

Executing shell commands with bash
Reading User Input

#!/bin/bash
 
echo -e "Hi, please type the word: \c "
read  word
echo "The word you entered is: $word"
echo -e "Can you please enter two words? "
read word1 word2
echo "Here is your input: \"$word1\" \"$word2\""
echo -e "How do you feel about bash scripting? "
# read command now stores a reply into the default build-in variable $REPLY
read
echo "You said $REPLY, I'm glad to hear that! "
echo -e "What are your favorite colours ? "
# -a makes read command to read into an array
read -a colours
echo "My favorite colours are also ${colours[0]}, ${colours[1]} and ${colours[2]}:-)" 

Reading User Input with bash

#!/bin/bash
 
echo -e "Hi, please type the word: \c "
read  word
echo "The word you entered is: $word"
echo -e "Can you please enter two words? "
read word1 word2
echo "Here is your input: \"$word1\" \"$word2\""
echo -e "How do you feel about bash scripting? "
# read command now stores a reply into the default build-in variable $REPLY
read
echo "You said $REPLY, I'm glad to hear that! "
echo -e "What are your favorite colours ? "
# -a makes read command to read into an array
read -a colours
echo "My favorite colours are also ${colours[0]}, ${colours[1]} and ${colours[2]}:-)" 

Flask Python API library

15:24 Posted by SRE Hacks , , , , , , , , , No comments
Eager to get started? This page gives a good introduction to Flask. It assumes you already have Flask installed.

Requirements

If you have a computer that runs Python then you are probably good to go. The tutorial application should run just fine on  Linux. Unless noted, the code presented in these articles has been tested against Python 2.7 and 3.4, though it will likely be okay if you use a newer 3.x release.

You should use pip to install and deploy Flask. To install Flask generally on your system, give this command on the command prompt:
       pip install flask
Note that you can also specify the Flask version, should you need (you won't need this one):
       pip install flask=0.10
This should install Flask and everything it depends on. Don't be surprised if you find it has installed several Python modules as a result.

Flask should now be installed! Now, time to create our first app.

A Minimal Application

A minimal Flask application looks something like this:
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'
So what did that code do?
  1. First we imported the Flask class. An instance of this class will be our WSGI application.
  2. Next we create an instance of this class. The first argument is the name of the application’s module or package. If you are using a single module (as in this example), you should use __name__ because depending on if it’s started as application or imported as module the name will be different ('__main__' versus the actual import name). This is needed so that Flask knows where to look for templates, static files, and so on. For more information have a look at the Flask documentation.
  3. We then use the route() decorator to tell Flask what URL should trigger our function.
  4. The function is given a name which is also used to generate URLs for that particular function, and returns the message we want to display in the user’s browser.
Just save it as hello.py or something similar. Make sure to not call your application flask.py because this would conflict with Flask itself.
To run the application you can either use the flask command or python’s -m switch with Flask. Before you can do that you need to tell your terminal the application to work with by exporting the FLASK_APP environment variable:
$ export FLASK_APP=hello.py
$ flask run
 * Running on http://127.0.0.1:5000/
If you are on Windows you need to use set instead of export.
Alternatively you can use python -m flask:
$ export FLASK_APP=hello.py
$ python -m flask run
 * Running on http://127.0.0.1:5000/
This launches a very simple builtin server, which is good enough for testing but probably not what you want to use in production.
Now head over to http://127.0.0.1:5000/, and you should see your hello world greeting.
Externally Visible Server
If you run the server you will notice that the server is only accessible from your own computer, not from any other in the network. This is the default because in debugging mode a user of the application can execute arbitrary Python code on your computer.
If you have the debugger disabled or trust the users on your network, you can make the server publicly available simply by adding --host=0.0.0.0 to the command line:
flask run --host=0.0.0.0
This tells your operating system to listen on all public IPs.

What to do if the Server does not Start

In case the python -m flask fails or flask does not exist, there are multiple reasons this might be the case. First of all you need to look at the error message.

Old Version of Flask

Versions of Flask older than 0.11 use to have different ways to start the application. In short, the flask command did not exist, and neither did python -m flask.

Invalid Import Name

The FLASK_APP environment variable is the name of the module to import at flask run. In case that module is incorrectly named you will get an import error upon start (or if debug is enabled when you navigate to the application). It will tell you what it tried to import and why it failed.
The most common reason is a typo or because you did not actually create an app object.

Debug Mode

(Want to just log errors and stack traces? )
The flask script is nice to start a local development server, but you would have to restart it manually after each change to your code. That is not very nice and Flask can do better. If you enable debug support the server will reload itself on code changes, and it will also provide you with a helpful debugger if things go wrong.
To enable debug mode you can export the FLASK_DEBUG environment variable before running the server:
$ export FLASK_DEBUG=1
$ flask run
(On Windows you need to use set instead of export).
This does the following things:
  1. it activates the debugger
  2. it activates the automatic reloader
  3. it enables the debug mode on the Flask application.
Attention
Even though the interactive debugger does not work in forking environments (which makes it nearly impossible to use on production servers), it still allows the execution of arbitrary code. This makes it a major security risk and therefore it must never be used on production machines.

GDB and it's use

15:15 Posted by SRE Hacks , , , , , , No comments

What is GDB?

GDB, the GNU Project debugger, allows you to see what is going on `inside' another program while it executes -- or what another program was doing at the moment it crashed.
GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act:
  • Start your program, specifying anything that might affect its behavior.
  • Make your program stop on specified conditions.
  • Examine what has happened, when your program has stopped.
  • Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.
The program being debugged can be written in Ada, C, C++, Objective-C, Pascal (and many other languages). Those programs might be executing on the same machine as GDB (native) or on another machine (remote). GDB can run on most popular UNIX and Microsoft Windows variants.

GDB is a well known debugger library for  c and c++.  GDB is usually use for large and distributed system debugging. GDB is command line tool and has almost all function that you use in debugging using IDE.
I have explain some basic and useful methods of GDB that help up in learning GDB and debugging your programs
Installation

Before you go for installation first check either GDB is already installed or not. You check using this command
gd -help

I GDB is already installed on your machine, it shows all the available options of GDB.
If not installed it show message to install GDB.  Then you install it manually but first check these prerequisites
  • An ANSI-compliant C compiler (gcc is recommended - note that gdb can debug codes generated by other compilers)
  • 115 MB of free disk space is required on the partition on which you're going to build gdb.

  • 20 MB of free disk space is required on the partition on which you're going to install gdb.
Now run this command to install GDB
sudo apt-get install gdb

 GDB Command Line Arguments:
Starting GDB:
  • gdb name-of-executable
  • gdb -e name-of-executable -c name-of-core-file
  • gdb name-of-executable --pid=process-id
    Use ps -auxw to list process id's: Attach to a process already running:
    [prompt]$ ps -auxw | grep myapp
    user1     2812  0.7  2.0 1009328 164768 ?      Sl   Jun07   1:18 /opt/bin/myapp
    [prompt]$ gdb /opt/bin/myapp 2812
    OR
    [prompt]$ gdb /opt/bin/myapp --pid=2812
                
Command line options: (version 6. Older versions use a single "-")
Option Description
--help
-h
List command line arguments
--exec=file-name
-e file-name
Identify executable associated with core file.
--core=name-of-core-file
-c name-of-core-file
Specify core file.
--command=command-file
-x command-file
File listing GDB commands to perform. Good for automating set-up.
--directory=directory
-d directory
Add directory to the path to search for source files.
--cd=directory Run GDB using specified directory as the current working directory.
--nx
-n
Do not execute commands from ~/.gdbinit initialization file. Default is to look at this file and execute the list of commands.
--batch -x command-file Run in batch (not interactive) mode. Execute commands from file. Requires -x option.
--symbols=file-name
-s file-name
Read symbol table from file file.
--se=file-name Use FILE as symbol file and executable file.
--write Enable writing into executable and core files.
--quiet
-q
Do not print the introductory and copyright messages.
--tty=device Specify device for running program's standard input and output.
--tui Use a terminal user interface. Console curses based GUI interface for GDB. Generates a source and debug console area.
--pid=process-id
-p process-id
Specify process ID number to attach to.
--version Print version information and then exit.

GDB Commands:
Commands used within GDB:





Command Description
help List gdb command topics.
help topic-classes List gdb command within class.
help command Command description.
eg help show to list the show commands
apropos search-word Search for commands and command topics containing search-word.
info args
i args
List program command line arguments
info breakpoints List breakpoints
info break List breakpoint numbers.
info break breakpoint-number List info about specific breakpoint.
info watchpoints List breakpoints
info registers List registers in use
info threads List threads in use
info set List set-able option
Break and Watch
break funtion-name
break line-number
break ClassName::functionName
Suspend program at specified function of line number.
break +offset
break -offset
Set a breakpoint specified number of lines forward or back from the position at which execution stopped.
break filename:function Don't specify path, just the file name and function name.
break filename:line-number Don't specify path, just the file name and line number.
break Directory/Path/filename.cpp:62
break *address Suspend processing at an instruction address. Used when you do not have source.
break line-number if condition Where condition is an expression. i.e. x > 5
Suspend when boolean expression is true.
break line thread thread-number Break in thread at specified line number. Use info threads to display thread numbers.
tbreak Temporary break. Break once only. Break is then removed. See "break" above for options.
watch condition Suspend processing when condition is met. i.e. x > 5
clear
clear function
clear line-number
Delete breakpoints as identified by command option.
Delete all breakpoints in function
Delete breakpoints at a given line
delete
d
Delete all breakpoints, watchpoints, or catchpoints.
delete breakpoint-number
delete range
Delete the breakpoints, watchpoints, or catchpoints of the breakpoint ranges specified as arguments.
disable breakpoint-number-or-range
enable breakpoint-number-or-range
Does not delete breakpoints. Just enables/disables them.
Example:
Show breakpoints: info break
Disable: disable 2-9
enable breakpoint-number once Enables once
continue
c
Continue executing until next break point/watchpoint.
continue number Continue but ignore current breakpoint number times. Usefull for breakpoints within a loop.
finish Continue to end of function.
Line Execution
step
s
step number-of-steps-to-perform
Step to next line of code. Will step into a function.
next
n
next number
Execute next line of code. Will not enter functions.
until
until line-number
Continue processing until you reach a specified line number. Also: function name, address, filename:function or filename:line-number.
info signals
info handle
handle SIGNAL-NAME option
Perform the following option when signal recieved: nostop, stop, print, noprint, pass/noignore or nopass/ignore
where Shows current line number and which function you are in.
Stack
backtrace
bt
bt inner-function-nesting-depth
bt -outer-function-nesting-depth
Show trace of where you are currently. Which functions you are in. Prints stack backtrace.
backtrace full Print values of local variables.
frame
frame number
f number
Show current stack frame (function where you are stopped)
Select frame number. (can also user up/down to navigate frames)
up
down
up number
down number
Move up a single frame (element in the call stack)
Move down a single frame
Move up/down the specified number of frames in the stack.
info frame List address, language, address of arguments/local variables and which registers were saved in frame.
info args
info locals
info catch
Info arguments of selected frame, local variables and exception handlers.
Source Code
list
l
list line-number
list function
list -
list start#,end#
list filename:function
List source code.
set listsize count
show listsize
Number of lines listed when list command given.
directory directory-name
dir directory-name
show directories
Add specified directory to front of source code path.
directory Clear sourcepath when nothing specified.
Machine Language
info line
info line number
Displays the start and end position in object code for the current line in source.
Display position in object code for a specified line in source.
disassemble 0xstart 0xend Displays machine code for positions in object code specified (can use start and end hex memory values given by the info line command.
stepi
si
nexti
ni
step/next assembly/processor instruction.
x 0xaddress
x/nfu 0xaddress
Examine the contents of memory.
Examine the contents of memory and specify formatting.
  • n: number of display items to print
  • f: specify the format for the output
  • u: specify the size of the data unit (eg. byte, word, ...)
Example: x/4dw var
Examine Variables
print variable-name
p variable-name
p file-name::variable-name
p 'file-name'::variable-name
Print value stored in variable.
p *array-variable@length Print first # values of array specified by length. Good for pointers to dynamicaly allocated memory.
p/x variable Print as integer variable in hex.
p/d variable Print variable as a signed integer.
p/u variable Print variable as a un-signed integer.
p/o variable Print variable as a octal.
p/t variable
x/b address
x/b &variable
Print as integer value in binary. (1 byte/8bits)
p/c variable Print integer as character.
p/f variable Print variable as floating point number.
p/a variable Print as a hex address.
x/w address
x/4b &variable
Print binary representation of 4 bytes (1 32 bit word) of memory pointed to by address.
ptype variable
ptype data-type
Prints type definition of the variable or declared variable type. Helpful for viewing class or struct definitions while debugging.
GDB Modes
set gdb-option value Set a GDB option
set logging on
set logging off
show logging
set logging file log-file
Turn on/off logging. Default name of file is gdb.txt
set print array on
set print array off
show print array
Default is off. Convient readable format for arrays turned on/off.
set print array-indexes on
set print array-indexes off
show print array-indexes
Default off. Print index of array elements.
set print pretty on
set print pretty off
show print pretty
Format printing of C structures.
set print union on
set print union off
show print union
Default is on. Print C unions.
set print demangle on
set print demangle off
show print demangle
Default on. Controls printing of C++ names.
Start and Stop
run
r
run command-line-arguments
run < infile > outfile
Start program execution from the beginning of the program. The command break main will get you started. Also allows basic I/O redirection.
continue
c
Continue execution to next break point.
kill Stop program execution.
quit
q
Exit GDB debugger.

 GDB Operation:


  • Compile with the "-g" option (for most GNU and Intel compilers) which generates added information in the object code so the debugger can match a line of source code with the step of execution.
  • Do not use compiler optimization directive such as "-O" or "-O2" which rearrange computing operations to gain speed as this reordering will not match the order of execution in the source code and it may be impossible to follow.
  • control+c: Stop execution. It can stop program anywhere, in your source or a C library or anywhere.
  • To execute a shell command: ! command
    or shell command
  • GDB command completion: Use TAB key
    info bre + TAB will complete the command resulting in info breakpoints
    Press TAB twice to see all available options if more than one option is available or type "M-?" + RETURN.
  • GDB command abreviation:
    info bre + RETURN will work as bre is a valid abreviation for breakpoints

Pandas Python Library

pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with structured (tabular, multidimensional, potentially heterogeneous) and time series data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. It is already well on its way toward this goal.
pandas is well suited for many different kinds of data:
  • Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet
  • Ordered and unordered (not necessarily fixed-frequency) time series data.
  • Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels
  • Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure
The two primary data structures of pandas, Series (1-dimensional) and DataFrame (2-dimensional), handle the vast majority of typical use cases in finance, statistics, social science, and many areas of engineering. For R users, DataFrame provides everything that R’s data.frame provides and much more. pandas is built on top of NumPy and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.
Here are just a few of the things that pandas does well:

  • Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data
  • Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects
  • Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let Series, DataFrame, etc. automatically align the data for you in computations
  • Powerful, flexible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data
  • Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into DataFrame objects
  • Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
  • Intuitive merging and joining data sets
  • Flexible reshaping and pivoting of data sets
  • Hierarchical labeling of axes (possible to have multiple labels per tick)
  • Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving / loading data from the ultrafast HDF5 format
  • Time series-specific functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc.

Data Structures

pandas introduces two new data structures to Python - Series and DataFrame, both of which are built on top of NumPy (this means it's fast).
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
pd.set_option('max_columns', 50)
%matplotlib inline


DataFrame

A DataFrame is a tablular data structure comprised of rows and columns, akin to a spreadsheet, database table, or R's data.frame object. You can also think of a DataFrame as a group of Series objects that share an index (the column names).
For the rest of the tutorial, we'll be primarily working with DataFrames.

Reading Data

To create a DataFrame out of common Python data structures, we can pass a dictionary of lists to the DataFrame constructor.
Using the columns parameter allows us to tell the constructor how we'd like the columns ordered. By default, the DataFrame constructor will order the columns alphabetically (though this isn't the case when reading from a file - more on that next).
data = {'year': [2010, 2011, 2012, 2011, 2012, 2010, 2011, 2012],
        'team': ['Bears', 'Bears', 'Bears', 'Packers', 'Packers', 'Lions', 'Lions', 'Lions'],
        'wins': [11, 8, 10, 15, 11, 6, 10, 4],
        'losses': [5, 8, 6, 1, 5, 10, 6, 12]}
football = pd.DataFrame(data, columns=['year', 'team', 'wins', 'losses'])
football
year team wins losses
0 2010 Bears 11 5
1 2011 Bears 8 8
2 2012 Bears 10 6
3 2011 Packers 15 1
4 2012 Packers 11 5
5 2010 Lions 6 10
6 2011 Lions 10 6
7 2012 Lions 4 12

Much more often, you'll have a dataset you want to read into a DataFrame. Let's go through several common ways of doing so.

Thursday, 2 February 2017

Contrail test and build commands guide

If you want to test some new feature in Opencontrail or want to test your current environment. You run test using scons commands. Some test are module specific and some are file specific. You may run then as you need. Also if you change some code in Opencontrail you need to re-built Opencontrail. Here are some useful and basic commands to re-build files, schema and tests. Please also read some important notes in this blog, that might help you in working with Opencontrail.

For compilation

Use this command to compile any file in controller.

       
 sudo scons src/control-node/  co=1 -u
       
 


If that doesn't work in your environment use this

       
 sudo scons control-node  co=1 -u
       
 


Your compiled file will be placed in contrail/build/debug/ folder. Then you run control service from debug which is currently running from contrail/build/production/.
Go to contrail screen and select control service. Stop control service using Ctrl + c and then run this in same screen service where you stop control service

       
  sudo /opt/stack/contrail/build/debug/control-node/contrail-control --conf_file /etc/contrail/contrail-control.conf & echo $! >/opt/stack/status/contrail/control.pid; fg || echo "control failed to start" | tee "/opt/stack/status/contrail/control.failure"

       
 


Change in Schema

If you make changes in any schema *.xsd file. You must first run this command to generate api's and other utility tools used by contrail

       
 scons controller/src/api-lib

       
 


Then run this command to check either previous command generate code or not

       
 grep bgp-origin build/debug/api-lib/vnc_api/gen/*

       
 


Note: Run stack.sh to compile all environment. It is compulsory to run stack.sh after changes in schema



Contrail test commands


In contrail you run test as a whole or by module or individually

Complete Contrail test

If you want to run all test in contrail you run this command (It may take 6-8 hours)

       
 scons test

       
 


If you want to ignore error and run test until end, use -i flag with this command

       
 scons -i test

       
 


Module wise test


If you want to run test of specific module test, use this command

       
 scons -u --optimization=production controller/src/bgp:test

       
 


OR

       
 scons -u -i --optimization=production controller/src/bgp:test

       
 


Individual Test


If you want to run test of specific file, you first compile that file and then manually run it's object file.

Compile that file using this command

       
 scons -u --optimization=production src/bgp:[test_file_name]

       
 


Then go to contrail/build/production/bgp/test , and run using this command

       
 ./[test_file_name]

       
 


If any error occurred while running test, you can see log file of that file in contrail/build/production/bgp/test with name [test_file_name].log

Note: All Scons command must run in /opt/stack/contrail folder
Note: To write result of Scons command in file, run command using 'Command > file_name.txt'

Saturday, 28 January 2017

Solution for ltcmalloc in Linux based system

14:57 Posted by SRE Hacks , , 1 comment
ltcmalloc is a library used in Linux based systems. This library is used for memory allocation.
You can find this library files in /usr/lib/.
But some time we get an error cannot find -ltcmalloc
Here is the screen shot of error


This is because of missing ltcmalloc in your system.
I run Opencontrail unit test and get this error


I automate the solution of this you can use the script bellow or use manual method given below

Automated Solution:

Put this code in .sh file and run it as root (using sudo)
       
command -v git >/dev/null 2>&1 || { echo >&2 "I require git but it's not installed.  Aborting."; exit 1; }
git clone https://github.com/imranhassanabdi/tcmalloc-files.git 
sudo cp -r tcmalloc-files/tc-malloc/* /usr/lib/.
sudo cp tcmalloc-files/libunwind.so.8 /usr/lib/x86_64-linux-gnu/
sudo rm -r tcmalloc-files/
sudo cd /usr/lib/
sudo ln -s libtcmalloc_and_profiler.so.4.1.2 libtcmalloc_and_profiler.so.4
sudo ln -s libtcmalloc_and_profiler.so.4.1.2 libtcmalloc_and_profiler.so
sudo ln -s libtcmalloc_debug.so.4.1.2 libtcmalloc_debug.so.4
sudo ln -s libtcmalloc_debug.so.4.1.2 libtcmalloc_debug.so
sudo ln -s libtcmalloc_minimal_debug.so.4.1.2 libtcmalloc_minimal_debug.so.4
sudo ln -s libtcmalloc_minimal_debug.so.4.1.2 libtcmalloc_minimal_debug.so
sudo ln -s libtcmalloc_minimal.so.4.1.2 libtcmalloc_minimal.so.4
sudo ln -s libtcmalloc_minimal.so.4.1.2 libtcmalloc_minimal.so
sudo ln -s libtcmalloc.so.4.1.2 libtcmalloc.so.4
sudo ln -s libtcmalloc.so.4.1.2 libtcmalloc.so
sudo ldconfig
       
 


Manual Solution:


  • Run this command in /usr/lib


       

            ll | grep ltcmaloc

       
 

  • If you get nothing means you have missing ltcmalloc files
  • Next you download ltcmalloc files, I have uploaded all files here you can clone it
After clone run these commands
  • Copy all files in tc-malloc to /usr/lib using this command

       
            sudo cp -r tcmalloc-files/tc-malloc/* /usr/lib/.
       
 

  • Then copy the other file in clone directory

       
            sudo cp tcmalloc-files/libunwind.so.8 /usr/lib/x86_64-linux-gnu/
       
 

  • Then run these commands in sequence to make linking between files

       
            sudo ln -s libtcmalloc_and_profiler.so.4.1.2 libtcmalloc_and_profiler.so.4
     sudo ln -s libtcmalloc_and_profiler.so.4.1.2 libtcmalloc_and_profiler.so
     sudo ln -s libtcmalloc_debug.so.4.1.2 libtcmalloc_debug.so.4
     sudo ln -s libtcmalloc_debug.so.4.1.2 libtcmalloc_debug.so
     sudo ln -s libtcmalloc_minimal_debug.so.4.1.2 libtcmalloc_minimal_debug.so.4
     sudo ln -s libtcmalloc_minimal_debug.so.4.1.2 libtcmalloc_minimal_debug.so
     sudo ln -s libtcmalloc_minimal.so.4.1.2 libtcmalloc_minimal.so.4
     sudo ln -s libtcmalloc_minimal.so.4.1.2 libtcmalloc_minimal.so
     sudo ln -s libtcmalloc.so.4.1.2 libtcmalloc.so.4
     sudo ln -s libtcmalloc.so.4.1.2 libtcmalloc.so
            sudo ldconfig
       
 

  • Now check that library install successfully using this command

       
            ldconfig -p | grep tcmalloc
       
 


If it shows result like this

       
libtcmalloc_minimal_debug.so.4 (libc6,x86-64) => /usr/lib/libtcmalloc_minimal_debug.so.4
libtcmalloc_minimal_debug.so (libc6,x86-64) => /usr/lib/libtcmalloc_minimal_debug.so
libtcmalloc_minimal.so.4 (libc6,x86-64) => /usr/lib/libtcmalloc_minimal.so.4
libtcmalloc_minimal.so (libc6,x86-64) => /usr/lib/libtcmalloc_minimal.so
libtcmalloc_debug.so.4 (libc6,x86-64) => /usr/lib/libtcmalloc_debug.so.4
libtcmalloc_debug.so (libc6,x86-64) => /usr/lib/libtcmalloc_debug.so
libtcmalloc_and_profiler.so.4 (libc6,x86-64) => /usr/lib/libtcmalloc_and_profiler.so.4
libtcmalloc_and_profiler.so (libc6,x86-64) => /usr/lib/libtcmalloc_and_profiler.so
libtcmalloc.so.4 (libc6,x86-64) => /usr/lib/libtcmalloc.so.4
libtcmalloc.so (libc6,x86-64) => /usr/lib/libtcmalloc.so
       
 


Then you are successful in copying filing and linking these files.

Wednesday, 8 June 2016

Libvirt objects state

10:48 Posted by SRE Hacks No comments
Libvirt

libvirt is an open source API, daemon and management tool for managing platform virtualization. It can be used to manage KVM, Xen, VMware ESX, QEMU and other virtualization technologies. These APIs are widely used in the orchestration layer of hypervisors in the development of a cloud-based solution.

To avoid ambiguity about the terms used, here are the definitions for some of the specific concepts used in libvirt documentation:
  • a node is a single physical machine
  • an hypervisor is a layer of software allowing to virtualize a node in a set of virtual machines with possibly different configurations than the node itself
  • a domain is an instance of an operating system (or subsystem in the case of container virtualization) running on a virtualized machine provided by the hypervisor
Libvirt distinguishes between two different types of domains: transient and persistent.
  • Transient domains only exist until the domain is shutdown or when the host server is restarted.
  • Persistent domains last indefinitely.


Libvirt States

There are five states of Instance (virtual machine) in libvirt. They are as following

  • Undefined
  • Defined
  • Runing
  • Paused
  • Saved

  • Undefined - This is a baseline state. Libvirt does not know anything about domains in this state because the domain hasn't been defined or created yet.
  • Defined or Stopped - The domain has been defined, but it's not running. This state is also called stopped. Only persistent domains can be in this state. When a transient domain is stopped or shut down, it ceases to exist.
  • Running - The domain has been created and started either as transient or persistent domain. Either domain in this state is being actively executed on the node's hypervisor.
  • Paused - The domain execution on hypervisor has been suspended. Its state has been temporarily stored until it is resumed. The domain does not have any knowledge whether it was paused or not. If you are familiar with processes in operating systems, this is the similar.
  • Saved - Similar to the paused state, but the domain state is stored to persistent storage. Again, the domain in this state can be restored and it does not notice that any time has passed.

Thursday, 2 June 2016

Libvirt and event based vm states

Now a days virtualization is one of the most common and fastest growing field in computer science. Basically now world move from physical and costly hardware to software based. Like if we need new machine we go to market and buy the new one, but virtualization made it easy, now we only spawn new machine with just few clicks.

So their is need for some tool to manage virtualization, Libvirt is an opensource daemon used for managing virtualization. It can manage many virtualization technologies like KVM, XEN, VMWARE ESX, QEMU.

Livirt has many use cases. I worked on a use case to detect change in vm states on event bases.
Open stack on back end use libvirt for virtualization. What if we want to detect any change in virtual machine states??? First method is you continuously check your vm by using ssh and polling, but it's bad approach. Libvirt allows you to detect vm current state. Libvirt developers writes a python plugin that detect any change in vm state and show it to you, without polling and any other over head on server. You can get agent from here. It is opensource, so you can modify it as you can.

Libvirt has 5 states for any vm


  • Undefined
  • Defined
  • Runing
  • Paused
  • Saved

The agent get state from libvirt whenever any of these event occurred and show details. Agent register itself in libvirt and get state on every event change. The agent show many details some of them are
  • Name of instance (openstack name, created by openstack)
  • UUID of instance
  • Event name
    • Defined,
    • Undefined
    • Started
    • Suspended
    • Resumed
    • Stopped
    • Shutdown

  • Event Detail (Detail of event to clarify action)
    • Added
    • Updated
    • Removed
    • Booted
    • Migrated
    • Restored
    • Snapshot
    • Wakeup
    • Paused
    • Migrated
    • IOError
    • Watchdog
    • Restored
    • Snapshot
    • Unpaused
    • Migrated
    • Snapshot
    • Shutdown
    • Destroyed
    • Crashed
    • Migrated
    • Saved
    • Failed
    • Snapshot
    • Finished