sed quit operator and seq operator

The seq operator in shell is to generate sequential numbers.As you can see.

 acer@ubuntu:~$ seq 10
1
2
3
4
5
6
7
8
9
10

And the quit operator in sed is mainly used to quit when aa particular match or substitution occurs.

acer@ubuntu:~$ seq 10 | sed '4 q'
1
2
3
4
acer@ubuntu:~$

As you may see from the above example the ouput is stopped once the match occurs.

substitute within a range

In this case we will see how we can use sed command within a particular range

for example we try to substitute the words inbetween wonderful and marvellous.
leaving behind the first and last word.

Content of the file to be modified:

acer@ubuntu:~$ cat ggg
hello life
wonderful
great
marvellous
awesome

OUTPUT with execution:

acer@ubuntu:~$ sed '/wonderful/, /marvellous/ s/.*/modified/' ggg
hello life
modified
modified
modified
awesome
acer@ubuntu:~$

removing newline character in shell and perl

 To my knowledge there are two ways to remove "\n" in shell:

1)using formatted printing in awk
2)using truncate(tr) option in shell


This can be further clarified by the below example:

acer@ubuntu:~$ echo "hello"
hello
acer@ubuntu:~$ echo "hello" | awk '{printf ("%s",$1) }'
helloacer@ubuntu:~$ echo "hello" | tr -d "\n"
helloacer@ubuntu:~$

As you may see in the first case there is a default "\n" with echo
But in the second and third case it is removed using the above said methods.


In perl to remove the "\n" we have a separate function chomp:

The way to use the function in perl is

acer@ubuntu:~$ cat kar.pl
#/usr/bin/perl
$dd="hello\n";
chomp($dd);
print "$dd";
acer@ubuntu:~$ perl kar.pl
helloacer@ubuntu:~$

awk variables

In awk there are many default variables which can be used .
We will see each of them with examples:
acer@ubuntu:~$ cat > record1
John robinson
koren inc
phno:555555555


usha
yyy technologies
ph no:546456456


dddd
aaa technologies
ph no:44444444


acer@ubuntu:~$
consider the above record in which our requirement is to produce a single line for each record having the name with phone no.
In the above record ,the each record is separated by blank line.

For producing we need to
1)change the input record separator(RS) which is default to "\n" to blank line.
2)change the Field separator which is default to " " to  "\n"

acer@ubuntu:~$
acer@ubuntu:~$ cat separate.awk
BEGIN { FS="\n";RS="" }
{ print $1,$NF}
acer@ubuntu:~$ awk -f separate.awk record1
John robinson phno:555555555
usha ph no:546456456
dddd ph no:44444444

Here as you may see from the ouput the each record is converted into a single line.Here $NF refers to the last field in the record.

In the same way we can change the OFS and ORS and change the usual way the ouput is displayed .

In this we change the Output field separator(OFS) to "\t" from default separator space.
and the Output Record separator(ORS)  to "\n\n" from the default separator "\n"

Sample example illustrates the change better.
acer@ubuntu:~$ cat separate.awk
BEGIN { FS="\n";RS="";OFS="\t";ORS="\n\n" }
{ print $1,$NF}
acer@ubuntu:~$
acer@ubuntu:~$ awk -f separate.awk record1
John robinson    phno:555555555

usha    ph no:546456456

dddd    ph no:44444444

acer@ubuntu:~$

Regarding the usage of NR variable in awk.It just prints the record number.
Sample example:
acer@ubuntu:~$ cat ggg
hello life
wonderful
great
marvellous
awesome
acer@ubuntu:~$ awk '{print NR,$1 }' ggg
1 hello
2 wonderful
3 great
4 marvellous
5 awesome
acer@ubuntu:~$

NR can also be used in this way.
acer@ubuntu:~$ cat ggg
hello life
wonderful
great
marvellous
awesome
acer@ubuntu:~$ awk 'NR==1{print $0 ": i am in first row" }' ggg
hello life: i am in first row
acer@ubuntu:~$
"$0" usuage in awk is to print the complete record as from the below example:
acer@ubuntu:~$ awk '{print $0 }' ggg
hello life
wonderful
great
marvellous
awesome
acer@ubuntu:~$
Hope this gives a better idea about awk variables.

Kindly comment was the information useful.



The below table provides overview of all the awk variables available.

Variable	Represents
NR $0 NF $1-$n FS OFS RS ORS FILENAME	record number of current record the current record (as a single variable) number of fields in the current record fields in the current record input field seperator (default: SPACE or TAB) output field seperator (default: SPACE) input record seperator (default: NEWLINE) output record seperator (default: NEWLINE) name of the current input file

finding no of blank lines in file using awk in shell

This program prints the no of blank lines in a file using awk in shell

acer@ubuntu:~$
acer@ubuntu:~$
acer@ubuntu:~$ cat > ddd

dsdsd

sdsdsd


dsds
acer@ubuntu:~$ awk 'BEGIN { x=0 }
> /$^/{ x=x+1 } # for counting the no of blank lines
> END { print " no of blank lines in the file  " x }' ddd
 no of blank lines in the file  4
acer@ubuntu:~$



For detailed understanding of BEGIN,END you can see the AWK PROGRAMMING MODEL blog in this site.

Thanks.

Was it useful.Kindly put your comment.

AWK programming model

awk is usually input driven .i.e. it executes the command for the no of lines in the file supplied or the no of lines which is piped to the awk command.

For example:

acer@ubuntu:~$ cat > fff

ewrrwer

ewrw

wer

werwe

ewrwer^Z

[1]+  Stopped                 cat > fff

acer@ubuntu:~$ awk '{ print "hello world" } ' fff

hello world

hello world

hello world

hello world

hello world

acer@ubuntu:~$

Here the hello world is printed  5 times since fff has 5 lines.

There is a exception to it you can use BEGIN or END command which executes without waiting for the input provided by the file or the pipe.

In case of BEGIN command in awk executes before the input from the file is processed.

For example:
acer@ubuntu:~$ cat ggg
hello life
wonderful
great
marvellous
awesome
acer@ubuntu:~$
acer@ubuntu:~$ awk 'BEGIN { print "hello from begin"}  
> /wonderful/ { print "hello during file process" }'  ggg    
hello from begin 
hello during file process
acer@ubuntu:~$

PROCESS IN AWK passes through three simple steps:

It executes the BEGIN COMMAND before any input is read.
                                     |
It process the file or input for the no of lines in file.(main loop)
                                     |
It executes the END command after the main loop ends.

Here the BEGIN and END are optional.

And a note about the END command it does not get executed if there is no lines in the file or input and it waits for input unlike BEGIN command.

acer@ubuntu:~$ awk 'END { print "hello from end command" }'
it waits endlessly

When you provide a input , it executes
acer@ubuntu:~$ echo "hello" | awk 'END { print "hello from end command" }'
hello from end command
acer@ubuntu:~$

These in combination can be used for writing many useful commands.

For example : for now we will write a shell program to print the no of lines in the file

acer@ubuntu:~$ awk 'BEGIN { x=0 }
> { x=x+1 }   # for adding the no of lines
> END { print "no of lines in the file is" x }' ggg
no of lines in the file is5
acer@ubuntu:~$


Hope this clarifies your doubt about AWK PROGRAMMING MODEL.

Kindly comment

sprintf usuage in perl and C

Sprintf:
Perl :
SYNTAX:sprintf(FORMAT,LIST);

sprintf is to produce the formatted string based on the input string.
It is based on sprintf C library routine .

sprintf is useless if  you give the no of digits to be padded less than the no of digits in your variable.As you may see from the below example:

dd="456456456";
$rr=sprintf("%04d",$dd);
print("$rr"."\n");

Output will be:456456456

But if you give the no of digits less than the padding digits.In addition to the digits available the remaining will be padded.
$dd="2";
$rr=sprintf( "%08d",$dd);
printf("$rr"."\n");
Output:00000002
Here 7 zeros are padded with the available digit 2.

The format has many options.In this case we round the digits to 3 decimal point.
$ff="333.45454545";
$rr=sprintf("%.3f",$ff);
printf("$rr"."\n");
Output:333.455

C language:
sprintf has a provision to save the formatted string in the syntax itself.

SYNTAX:sprintf(saved string ,format,string to be formatted);

Here the change is the formatted string is saved in the syntax itself,instead of saving through equality condition.

#include <stdio.h>int main ()

{
  char buffer [50];
  int n, a=5, b=3;
  n=sprintf (buffer, "%d plus %d is %d", a, b, a+b);
  printf ("[%s] is a %d char long string\n",buffer,n);
  return 0;
}

Output:
[5 plus 3 is 8] is a 13 char long string

finding the memory size in Linux

To my knowledge there are three command to find the memory in Linux:

1)free command:

It is used to print the memory size in bytes by default

>> free
total used free shared buffers cached
Mem: 16438664 16294220 144444 0 235596 13380932
-/+ buffers/cache: 2677692 13760972

options:

-b - to print the bytes

-m - to print the information in MegaBytes

-k - to print the information in kilobytes.

>>free -m
total used free shared buffers cached
Mem: 16053 15953 100 0 214 13119
-/+ buffers/cache: 2619 13433
Swap: 9855 1 9854


2)Top command:

It is used to display the ongoing process in the Cpu and the memory used by the same.

>>Top
PID USERNAME THR PRI NICE SIZE RES SHR STATE TIME CPU COMMAND
4536 root 1 0 -20 37M 8696K 2204K sleep 189:47 1.60% mount
23308 karthik 1 15 0 18M 5948K 4392K sleep 0:00 1.60% vim
3754 root 1 16 0 10M 680K 584K sleep 3:03 0.20% vi


3)dmesg | grep ^Memory
This also gives the information about the available memory.

> dmesg | grep ^Memory
Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Memory: 16435064k/17563644k available (2577k kernel code, 339988k reserved, 1305k data, 212k init

awk unix command

awk is a unix command

Awk is used to get a particular pattern in a ouput

you can use this to print a particular pattern

To print the third column in the output using space as the delimiter
awk -F " " '{print $3}'

If you want to print the particular row and a particular column you can use like this

awk -F " " 'NA==3{print $2}'

-F is used as a the Field separator

awk print

Awk is used to get a particular pattern in a ouput

you can use this to print a particular pattern

To print the third column in the output using space as the delimiter
awk -F " " '{print $3}'

If you want to print the particular row and a particular column you can use like this

awk -F " " 'NA==3{print $2}'

Comparing NULL in C example

You can use the below code for comparing String with NULL in C

char * ff;
if (  ff != NULL ) );
{
printf("value isnot null");
}
else
{
printf ("value is null");
}

Comments in C and C++

In C

valid comments are when text are in between /*  .......*/
It can also be used across multiple lines.But it will not support nested comments.

ex:
/*  comments  */
or

/*   First line
Second line  */

It can also be used like:
/*  comments
*   comments
*  comments
*/

 new C compilers also supports inline comments    // .  but old compilers may give syntax error.
ex:
int c;  // for intializing

 C++  supports both type of comments inline comments // and /*  ...*/

strcmp and strncmp in C

strcmp function in c

int strcmp(char * str1,char * str2)
strcmp function  returns a number eitheir 0, positive integer or negative integer

Return vaues:

positive integer       - is returned if string str1 is greater than string str2

0                           - if both the strings are equal

negative integer      -  if string str1 is less than the string str2

In this case comparisons happen using unsigned integers.

for example:
char * i = "hello";
char * j="hello1"; 
printf("%d",strcmp(i,j));
output will be a negative integer

 strncmp in C:

The syntax all are same .but in this case there is an extra argument for setting how many characters you want to compare.

int strncmp(char *str1,char * str2, int );

Functionality all are same.But here int parameter is to set no of characters to compare in two strings.

for example:

char * i = "hello";

char * j="hello1"; 

printf("%d",strcmp(i,j,4));

output will be 0 as we compare only 4 characters and it is same in both the strings.

hosts.allow and hosts.deny in Linux

hosts.allow and hosts.deny files are mainly used for restricting the ssh to th editing  your server in which you are    
editing the files.

These files are maintained by the TCP wrappers

These are mainly used for securing your server or system from outside acccess.

The way it happens when a outside ip asks for permission to ssh to your system:
1)The Tcp wrapper first checks for entry in /etc/hosts.allow.If a entry is present ,it does not go to the /etc/hosts.deny.
2)If entry is not present ,then it goes for hosts.deny ->if entry is present , the ip is denied
                                                                                  if entry is not present ,the ip is allowed.

If the entry is not present in both hosts.allow and hosts.deny , the entry is allowed.

"=" condition in shell

uae -ne for number and "=" for string

Comparators in shell

In case of shell comparison one need to be very careful as one extra space may lead to the code throwing errors

For example :
if [$dd=1]
then
.....statement
fi

This is completely wrong as it will throw a "syntax error near unexpected token near then"
as there are three errors in this:
1)there must be minimum of one space between if and square bracket "["
2)For comparison in shell  ,there must be minimum of one space before and after the equal to sign"="
3)minimum of one space needed between 1 and closing bracket
giving more spaces will not cause error to happen.

proper code:

if  [ $dd = 1 ]
then
.....statement
fi

Shell assignments:

Ironic in the case of assignments when compared to comparison,is that there should not be any space before and after the equal-to sign("=")

FOR EXAMPLE:
Inside the shell file you provide like this:
gg  =  1

This will throw a critical error like "unary operater expected " and "command not found"
So it is must in Shell that the assignments must not have space before and after the "=" sign.
It should be:
gg=1

Checking not-equal condition in Shell:

you can use "-ne" or "!= "  can be used for numbers.

you can use "!=" for comparing strings are not equal.

Logical Operators in Shell:

and     -a  can be used

cond1 -a  cond2  -  True if both conditions are true.

ex: [  $gg -eq  1 -a $cc -eq  2 ]  .Maintain spaces here else you may get "  Too many arguments error"

or     -  o can be used

ex: [ $gg -eq 1 -o $cc -eq   2 ]

Checking files in Shell:

-f somefile True if somefile exists and is an ordinary file.

-d somefile True if somefile exists and is a directory.

-s somefile True if somefile contains data (the size is not zero).

-r somefile True if somefile is readable.

-w somefile True if somefile is writable.

-x somefile True if somefile is executable.

exporting variable in Shell

You can create your own environmental variables in shell

It is very simple just type export followed by your variable and the value.

For exporting a variable gg

export gg="hello"

echo $gg
hello
you will be able to see the content of the variable gg - "hello"

But it stays untill reboot.Once you reboot you have to set the variable again.

To make it permanent you can add it to /etc/profile to make the change visible for all users using your system or you can add the change to the ~/.bash_rc to make the change visible only for your shell.

Changing the prompt in Shell

For changing the prompt in shell

we need to export the PS1 variable

Its by default set to \$ and # for root

To change it  simply follow the below step:

you can change to many special characters.but commonly used are:

\$     -  shows the user prompt($) or root prompt(#) ,depending on which user you are.

\W   -  shows the current working directory base name.if the current working directory is "/home/karthik/perl" then it simply appears as "perl"

\w   -  Displays the full path of the current working directory.

\t    -   prints the current time in hours.minutes and seconds(for ex:9:30:40)

\s   - prints the current shell name.For bash shell it simply prints bash as prompt.

\u - in this case it prints the current username.

These options can also be used together.

For ex:
export PS1="\t \w \$"
makes the prompt as
[18:30:40 /home/karthik]$>

export PS1="\W"
makes the prompt as  if the full path is "/home/karthik/perl"
perl>

Files for configuring your shell

/etc/profile     
Sets up user information for every user.
It is executed when you first log in.
 It sets up the path as well as the environmental variables,the location of your mailbox and size                                                                            of your history files.
Finally,files under the /etc/profile.d are executed.

/etc/bashrc   
 Runs for every user who runs the bash shell.
 It sets a default prompt and may add one or more aliases.
But this file can be overridden by information in ~/.bashrc file.

~/.bash_profile 
It is run when the particular user logs in.
Used by the user to enter information that is specific to  his or her own use of the shell.
This file inturn calls the user's .bashrc file

~/.bashrc         
Contains the information specific to the particular user.
This is the best location to add environmenta  variables and your aliases so that shell      
 will pick up when you log in.
 
~/.bash_logout   
Executes when you log out of your bash shell.
 By default just simply clears the screen.(this is also for a particular user)

using chmod command (or) Setting File permissions in Linux,Shell or Unix

chmod -is used to change the permissions for a file.
when you type the name of the file


ls -ld  - you will be able to see the permissions provided for the file

there are three groups to which permissions can be provided:
o - others
g - user group
u -users

to set the permissions for the file use:
chmod command followed by the fiename.

there are three permissions namely read,write,execute which can be given to the users,user group and others

For providing permission to a file we must be owner of the file or root
read permssion -4
write permission-2
execute permission-1

to give all permissions to a particular group such as user
provide 7=4+2+1 and no permission to usergroup and others
chmod 700 filename
ls -ld
-rwx------ 1 chris sales 4983 jan 23 22:13 ch3
 ---   ---
 user  others
    ---
group
1nd)r - read
2nd)w - write
3nd)x - execute

Other way to do that:
a - all users
g - owner group
u - owner user
o - others

for permissions use:
r-read
x-execute
w-write

yu can either add or subtract the permissions:

assume the permission for the file to be rwxrwxrwx
permission change when each command is executed:
chmod a-w file     r-xr-xr-x
chmod o-x file     rwxrwxrw-
chmod go-rwx file  rwx------

assume the file to have a permission of ---------
chmod u+rw files   rw-------
chmod a+x files    --x--x--x
chmod ug+rw files  rw-rw----

to change the permissions for the entire directory we have to -R opton to recuresively change the permission for the entire directory

chmod -R 777 /tmp/hello/
this command willchange the perission for the entire hello directory to 777

visual mode in vi

These commands can be used only after you enable visual mode in vim editor

Shift+V - to enable the visual mode in vim

Then select the lines in the vi

Alt+> - Shift right the selected lines

Alt+< - Shift left the selected lines

c - change the selected lines

y - copy or yank the selected lines to the clipboard.

p - paste the copied lines from the clipboard to the Editor.

getenv in C

getenv in C

this is to get the environmental variable's value in Shell from C file

to store it to a variable use:

fp=getenv("PATH")

The following program will explain it more clearly


This program demonstrates how to get a value for environmental value in shell
and compare it with a predefined variable.

1 #include
2 #include
3 #include
4 int main()
5 {
6 char * cc;
7 cc = getenv("HOME");
8 printf("--->%s<--",cc);
9 if ( strcmp(cc,"/home/karthik") == 0 )
10 {
11 printf("yes the home is /home/karthik");
12 }
13 return 0;
15 }

Editing in vi

Many keywords ease the use of vi

Some of them are:

Ctrl+F - to move forward one page at a time

Ctrl+B - to move backward one page at a time

Ctrl+D - to move half page forward at a time

Ctrl+U - to move half page backward at a time

Ctrl+G - name of the file you are editting

G -last line of the file

1G - move to the first line of the file
--
this any line no you want to move to


H - want the cursor to move to the upper left corner of the screen

M - want to move the cursor to the middle left corner of the screen

L - want to move the cursor to the lower left corner of the screen

Ctrl+R - to Redo the changes in the file

Ctrl+U - to Undo the changes in the file

dw - delete the word

d$ - delete starting from where the cursor is to the end of the line

d0 - delete starting from where the cursor is to the beginning of the line


Editing using numbers:

5cl - delete the 5 characters following the cursor and make it in insert mode

12j - move the cursor down 5 lines

3dw - delete 3 words

substitution in vi:

:g/Local - find the occurence of Local globally and print the instances

:s/Local/Remote - replace the occurence in the current line

:g/Local/s//Remote - for replacing the first occurence

:g/Local/s//Remote/g - for replacing the entire occurence of Local

:g/Local/s//Remote/gp - for replacing and printing the changed occurence

comparators in perl and shell

SHELL:

KSH

you can use == or = to compare string

bash: if you use "==" will lead to syntax errors so be cautious,we have to use only "=" in this case.

Another speciality about = operator is it can be used for both string and numbers.

and can use -eq  for only comparing numbers

And -eq if used for compring strings,it will return syntax error.

PERL

it is just the opposite what you  use in shell

eq can be used to compare only string

= can be used for only numbers

Though = may seem to work for strings it will always return true.So be Cautious.

Connection refused(public key,password authentication,keyboard interactive)

This error comes when you are trying to login to a machine using ssh or telnet

This may be due to

1)the password you provide may be wrong

2)next chance may be the user account you are using to login to a remote machine can be locked.

for example:ssh karthik@123.45.66.77

this may throw this error
 In this case the karthik account in 123.45.66.77 can be locked.


This can be found out by checking /etc/shadow file in 123.45.66.77 .The karthik account in /etc/shadow file
can be mentioned with *LK* showing it is locked.

extracting .cpio file

syntax:

cat cpio_file | cpio -idmv
use this to extract .cpio file


This works fine.

configuring ssh in Ethernet Switch

Before configuring ssh module in Ethernet switch make sure that ssh modue is installed.

To check ssh module is installed :

show management

this command shows all the module installed in the Ethernet switch  :
if you find the ssh module not installed.Install it before configuring ssh.

After installing  exsshd module,do the following steps:

1)configure ssh2 keys
this is to generate the ssh2 keys

2)enable ssh2
this is to enable the ssh2 module in Ethernet switch.

3)show configuration exsshd 
This will show the configuration of ssh module installed in Ethernet switch.

For locking a account and unllocking

For Locking the account you can use

passwd -l {username}

For unlocking the account you can use

passwd -u {username}

Once the account is locked no person can login to the user via the locked account.

The change can be noticed in /etc/shadow's fie for the locked account.

The account can  be locked only by the superuser or any user with that  privelage.

For adding a role to a user account

You can use the command

usermod -R adminsecurity  anand

here anand is a user
and adminsecurity is the role added to the user anand

For adding the role to the user,you must be a super user(ROOT).

port 22 connection refused

if this errors while trying ssh
Check sshd daemon is running in the server machine

you can check this by logging into the server machine and trying

ps -ef | grep sshd


if you find the ssh daemon process you check port 22 is used by some other process 


netstat -tupn
or  
chkconfg --list sshd

Setting up your own ip address for the interface card

You can use set any ip-address for your IP-address
ifconfig  interface IP-address netmask 255.255.255.0 broadcast 192.168.10.255

ifconfig eth2 192.168.10.12 netmask 255.255.255.0 broadcast 192.168.10.255


For bringing up interface use
ifconfig eth2 up

For bringing down the interface use
ifconfig eth2 down

striicthostkeychecking in ssh

Stricthostkey option in ssh

this can be configured in /etc/ssh/ssh_config  or it can be given in the command mode by specifying
ssh -o StrictHostKeyChecking=no 123.33.44.45
        

no option:  in this case when stricthostkey option is set to no:
1)when host key for the connecting server is not in the $HOME/.ssh/known_hosts file,it is added to the file after asking the confirmation from the server.
2)when there is a host key mismatch,it simply connects after showing a warning.
3)not advisable as security is very low.

ask option is the default option:
1)In this case also,when host key for the connecting server is not in the $HOME/.ssh/known_hosts file,it is added to the file after asking the confirmation from the server
2)But when there is a host key mismatch,it denies the connection showing the place where the mismatch has happened.

on option:
1)it is the strictest mode and unfriendliest mode..
2)In this case,when host key for the connecting server is not in the $HOME/.ssh/known_hosts file it simply denies the connection.
2) when there is a host key mismatch,it denies the connection.

host-key mismatch error in ssh

Solution:
This error happens because the privatewtho key in server side and public key in client side is not matching.This is how  public key authentication happens in ssh.

If you want to ssh without key based authentication you can
1) first make the strictHostKeyChecking in your /etc/ssh/ssh_config to no
2)and then remove the particular host key of the server you are trying to ssh from the /etc/ssh/known_hosts file

NOW THE SSH WILL HAPPEN THOUGH WITHOUT MUCH SECURITY.

To ssh with security,you can follow the steps in the blog for public key authentication.