Saturday, October 26, 2013

Setup Hadoop 2.x (2.2.0) on Linux (Fedora19)

Prerequisite -

1- install Java (openjdk-7-jdk or oracle-jdk-7)
2- install openssh-server

Hadoop Installation Process

1- Add Hadoop Group and User
$ groupadd hadoop
$ useradd hduser -g hadoop
$ passwd hduser

2- After user is created, login using hduser and Setup SSH Certificate
$ ssh-keygen -t rsa -P 'some_text'

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ ssh localhost

3- Download hadoop 2.2and extract
$ wget http://www.trieuvan.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
$ tar xvfz hadoop-2.2.0.tar.gz



4- Setup Hadoop Environment Variables
$ vi ~/.bashrc
and add below lines at the end of the file.

#Hadoop variables
export JAVA_HOME=/soft_install/jdk1.7.0_05/
export HADOOP_INSTALL=/home/hduser/hadoop-2.2.0
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
###end of paste

$ cd /home/hduser/hadoop-2.2.0
$ vi hadoop-env.sh

create file hadoop-env.sh by adding below lines.

#modify JAVA_HOME
export JAVA_HOME= /soft_install/jdk1.7.0_05/
$ . ~/.bashrc

$ hadoop version
Hadoop 2.2.0
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
This command was run using /home/hduser/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar


Congratulation..! hadoop is installed.


Post-Installation Configuration for Hadoop

$ cd /home/hduser/hadoop-2.2.0/etc/hadoop
$ vi core-site.xml
#Paste following between <configuration>

<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
</property>

$ vi yarn-site.xml

#Paste following between <configuration>

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>


$ cp mapred-site.xml.template mapred-site.xml

$ vi mapred-site.xml
#Paste following between <configuration>

<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>

$ cd ~
$ mkdir -p mydata/hdfs/namdnode
$ mkdir -p mydata/hdfs/datanode
$ cd /home/hduser/hadoop-2.2.0/etc/hadoop
$ vi hdfs-site.xml

Paste following between <configuration> tag

<property>
   <name>dfs.replication</name>
   <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/mydata/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/mydata/hdfs/datanode</value>
 </property>

Format Namenode
$ hdfs namenode -format


Start Hadoop Service
$ start-dfs.sh
$ start-yarn.sh
$ jps
If everything successful then you will get following services.
8379 DataNode
9097 NodeManager
8805 ResourceManager
8202 NameNode
9253 Jps
8599 SecondaryNameNode


Wednesday, August 21, 2013

CSS Hacks...!

1- Below are possible ways to add style-sheet.

a) Internal Style Sheet --      <style type="text/css"> ......</style>
b) External Style Sheet --     <link rel="stylesheet" type="text/css" href="css/global.css">

or XHTML way

<link rel="stylesheet" type="text/css" href="css/mystylesheet.css">

c) Linking a Style Sheet using CSS -- <style type="text/css">@import url(css/mystylesheet.css);</style>

NOTE- import technique can slow your style sheets’ download speed   @www.stevesouders.com/blog/2009/04/09/dont-use-import/

d) Inline StyleSheet -- <h1 style="color: #FFFFFF;">


2- Selectors in CSS
Tag Selector       --    tag_name {  }
Class Selector    --   .class_name {  }
ID selector          --    #id {  }
Group Selector   --    tag, .class, #id, .class1 {   }
Universal Selector       --   * {  }
Descendent Selector   --   html  body  ul  li  a  {  }      


3- Pseudo-classes
a.link  -- any non-visited link
a.visited -- any link which is visited as per browser history
a.hover -- how to behave when visitor passes mouse over it
a.active -- how a link looks as visitor click


4- To Remove dotted border which appear while clicking on anchored text use

a:active, a:focus {outline:  none;ie-dummy: expression(this.hideFocus=true);}


5- Pseudo-element
:before -- add content preceding given element
            to put content before paragraph -- p.tip:before {content: "HOT TIP!" }
:after  -- add content after given element
:first-child -- select the first child
:focus -- visitor does something to indicate her attention to a web page element—usually by clicking or tabbing into it
( IE6 and IE7 do not support :before, :after and :focus pseudo element)

Monday, July 29, 2013

shell-script hacks


1- send attachment using shell script

(printf "%s\n" \
    "Subject: test" \
    "To: $USER" \
    "Content-Type: application/zip" \
    "Content-Disposition: attachment; filename=image.tar.gz" \
    "Content-Transfer-Encoding: base64" \
    "";
 base64 image.tar.gz) | /usr/sbin/sendmail "$EMAIL_ID"

Monday, June 17, 2013

JavaScript or Jquery hacks....!

1- If you want to redirect to some other url after some seconds; say 5 seconds here; than you need to do below.

 \$(window).load(function() { window.setTimeout(function() { window.location.href = "https://www.google.co.in"; },5000) });

Here function call at setTimeout is necessary because without it redaction will occure immediately.

NOTE-  Jquery load function is triggered when after loading the page in web-browser. Where as ready() function invoked when DOM is ready.

Below codes  won't work as expected.

document.ready(window.setTimeout(location.href = "https://www.google.co.in",5000));

OR

$(window).load(function() { window.setTimeout(window.location.href = "https://www.google.co.in",5000); });

Saturday, June 15, 2013

Wordpress hacks...

1- If below wordpress error in module installation
"
Unpacking the package…
Abort class-pclzip.php : Missing zlib extensions
"
just list the compiled modules with php with "php -m"
or
check with wordpress function which require zlib moulde "var_dump(function_exists("gzopen"));"

whether current php supports zlib or not.if not than recompile it with --with-zlib option.Second
should return true if zlib is one of the compiled module.

Below is one sample to manually configure and install php.

./configure --prefix=/soft_install/php5 --with-apxs2=/soft_install/apache2/bin/apxs --with-mysql
--enable-fastcgi --enable-fpm --with-zlib --with-bz2 --with-xml --with-zlib --enable-mbregex
--with-mhash --enable-xslt --enable-memcache --enable-zip --with-pcre-regex

make & make install

Tuesday, April 23, 2013

mysql hack -- some tips..!

1- Mysql engine selection

MySQL uses row-level locking for InnoDB tables to support simultaneous write access by multiple sessions, making them suitable for multi-user, highly concurrent, and OLTP applications. MySQL uses table-level locking for MyISAM, MEMORY, and MERGE tables, allowing only one session to update those tables at a time, making them more suitable for read-only, read-mostly, or single-user applications. " @http://dev.mysql.com/doc/refman/5.5/en/internal-locking.html

hack perl...! some tips

1- utf-8 encoding and decoding

      'lc' do not support utf-8 characters directly.Hence below solution

        use Encode qw(encode decode);
        $word_key=decode('utf-8',$word_key);
        $word_key=lc($word_key);
        $word_key=encode('utf-8',$word_key);
     

2- Wide character print error
     Please encode the data than print it.

3- unicode to perl character

     my $string = "Hello \x{263A}!\n";
    @arr= split //,$string;
     print "@arr";

4- Read a file which has mixed characters ( ASCII and Unicode codes in the form of $<4-hex-code>) and then convert them to lowercase

#!/usr/bin/perl
use utf8;
use Encode;
open $file,'<:utf8','sample';
while(<$file>) {
s/\$(\p{AHex}{4})/chr hex $1/eg;
#s/\$(....)/pack 'U4', $1/eg;   /// this wont work
$_=lc($_);
$_=encode('utf-8',$_);
print;
}

#!/usr/bin/perl
=pod
$line = 'i m gaurav $0040 pant';
$line =~ s/\$(\p{AHex}{4})/chr hex $1/eg;
print $line;
=cut
#$line = 'i m gaurav \u0040 pant';
$line = 'T\u1ea5t c\u1ea3';
$line =~ s/\\u(\p{AHex}{4})/chr hex $1/eg;
print $line;
~                 

Tuesday, March 12, 2013

Passwordless SSH

                                            
SUPPOSE CLIENT TRY TO CONNECT REMOTE SERVER.

cd $HOME [or just type cd and then enter]


the server HOME directory permission 750


chmod 700 .ssh
cd .ssh            # goto .ssh folder where below files exist


To generate id_rsa.pub and id_rsa execute below command. Press enter
if it asks for anything
[If id_rsa.pub already there no need to do this. ]
ssh-keygen -t rsa

chmod to below files on both servers.
chmod 600 authorized_keys
chmod 644 known_hosts
chmod 640 id_rsa.pub
chmod 600 id_rsa

hostkey located in: /etc/ssh/ssh_host_rsa_key.pub
To show fingerprint: ssh-keygen -l -f ssh_host_rsa_key.pub

Append remote server's hostkey to client's(local) known_hosts file
with IP as below 
[IP hostkey= comment_to_describe_key]:

Append client server's(Local) id_rsa.pub key to remote server's authorized_keys
as below.
ssh-rsa key= comment_to_describe_key 
Now connect to remote servers as below example:
ssh ruser@1.2.3.4

It should not prompt you any password
 
NOTE-- There may be slight changes from OS to OS as in some OS keys been copied automatically. 

Sunday, March 3, 2013

Some useful Linux utilities-- useful tips

1- Sync two folders (below syncing each second)
         'watch -n 1 rsync -avz source destination'
 It also take care of directory structure.Also can sync remote system directory without mounting it.

2- bzcat,bzless,zcat etc commands are very useful to display the contents of single large compressed file(in some GB). If  any-time such large file need to be processed than use any of these command and pipe output to the programme which will process it.


3- Run command at remote system

ssh gauravp@machine "\unix_command"
or
ssh -n gauravp@machine "unix_command"
or
echo "unix_command"|ssh gauravp@machine

Saturday, March 2, 2013

connect internet using wvdial in Linux

Change wvdial.conf for connect to different internet connection--netconnect,bsnl,airtel

to connect command is-- wvdial dialer_name.

For reliance net connect+ its mandatory to provide username & password.

Note- A slight modification on this file will work for other ISP or other countries too.

/etc/wvdial.conf

[Dialer netconnect]
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Stupid Mode = 1
Modem Type = Analog Modem
New PPPD = yes
Baud = 9600
ISDN = 0
Modem = /dev/ttyUSB0
Phone = #777
Username = ur_number
Password =  ur_number
Baud = 9600

[Dialer bsnl]
Init1 = ATE0V1
Init2 = AT+CFUN=6
Init3 = AT+CGDCONT=1,"IP","bsnlnet"
Stupid Mode = 1
Modem Type = USB Modem
Phone = *99#
Modem = /dev/ttyACM0
Username = { }
Password = { }
Baud = 460800

[Dialer airtel]
Init1 = ATZ
Init2 = at+cgdcont=1,"ip","airtelgprs.com"
Stupid Mode = 1
Modem Type = USB Modem
ISDN = 0
Phone = *99***1#
Modem = /dev/ttyUSB0
Username = a
Password = a
Baud = 9600

Saturday, February 16, 2013

Apache installation error and solution

ERROR 1-
libtool: install: error: cannot install `libaprutil-1.la' to a directory not ending /some_directory

Solution-  If ./config --prefix=/mydir didn't get set correctly the first time. Rerunning a failed apache make can be troublesome. You need to first do a make clean, and then retry with the correct --prefix

Saturday, February 2, 2013

Problems- upgrading Fedora 16 to 17

If you have existing Fedora 16 or greater in your system and want to upgrade to fedora 17 using "iso" image than below is the trick to add entry into fedora 16 or onward.

 Please add this entry into "/etc/grub.d/40_custom" file and run command "grub2-mkconfig".

menuentry "Fedora 17 Linux" {
        set root=(hd0,1)
        linux /boot/vmlinuz repo=hd:/dev/sda3:/
        initrd /boot/initrd.img
    }
Some details about above entry-
vmlinuz and initrd.img files has been copied into /boot directory of F16 or above from fedora 17.dvd.iso after loop mounting.This boot partition is in /dev/sda1. Hence the entry set root=(hd0,1).

"repo=hd:/dev/sda3:/" will tell that iso image is in harddisk's partition /dev/sda3 and in root directory.You can give directory structure if image resides into some directory structure.

NOTE- Fedora 17 onward there are more simpler mechanism available.




If someone face below error after upgrading fedora 16 to 17 than here is quick resolution of these problems

==> wvdial is throwing below error.
"--> Modem initialized.
wvdial: utils/wvtask.cc:409: static void WvTaskMan::_stackmaster(): Assertion `magic_number == -0x123678' failed.
Aborted (core dumped)"

please install "libwvstreams" package using yum.
yum -y update libwvstreams

Thursday, January 31, 2013

reset root password of mysql

If you are getting problem ins accessing mysql using root because you forgot password and getting below error.

error snapshot-
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES/NO)

Than  you can reset the root password using below.

"sudo dpkg-reconfigure mysql-server-<version-number>"

OR

/usr/bin/mysqld_safe --skip-grant-tables &
mysql --user=root mysql
update user set password=PASSWORD("mynewpassword") where User='root';
 
"with --skip-grant-table" it will allow root access without password.

Thursday, January 24, 2013

The 403 Forbidden error troubelshooting- Apach2

1- Check for URL errors and make sure you're specifying an actual web page file name and extension, not just a directory. Sometimes configuration may be to disallow directory browsing so a 403 Forbidden message when trying to display a folder instead of a specific page.

2- Clear the browser cookies etc.

3- A 403 status code indicates that the client cannot access the requested resource. It means the wrong username and password were sent in the request, or that the permissions on the server do not allow what was being asked.(If user level restrictions enabled/configured in .htaccess etc)

4- No default directory index page is present. Upload Index.html or Index.htm file. Directive DirectoryIndex defines the default index page name. Open your apache configuration file, find out default index file name, and upload the same file to directory:
DirectoryIndex index.html index.cgi index.pl index.php index.xhtml

5- Make sure the CGI script requested have executable permissions set on files.

6- Make sure you have permission to use .htaccess file for Apache web server. If Apache has overrides disabled than apache will bump back with a 403 error.

7- Make sure correct directory permissions are set on directory:
Alias /alias_name  /path_to_webdir

<Directory "/alias_name">
        Options +Indexes FollowSymLinks +ExecCGI
        AllowOverride AuthConfig FileInfo
        Order allow,deny
        Allow from all
</Directory>

8- Make sure the underlying file system permissions allow the User/Group under which Apache is running to access the necessary files.Execute permission on directories so that to go through the directories.


Friday, January 18, 2013

Transparent setting of terminal is not showing actual windows

if the transparent settings in terminal just shows the desktop background and not the actual windows that are opened behind the terminal just fire below command

gconftool-2 -s '/apps/metacity/general/compositing_manager' --type bool true

and check the progress of running tasks without switching application windows.

Friday, January 4, 2013

perl CGI Hello world troubelshooting

To configure Apache2 with perl please make change in httpd.conf as below.

1- LoadModule cgid_module modules/mod_cgid.so
    Above module must be enabled.
2- Alias name for the directory containing the CGI script must be defined.
    ScriptAlias /cgi-bin/ "/usr/local/apache2/cgi-bin/"
3- Below part of httpd.con should be like it.
<Directory "/usr/local/apache2/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
    Order allow,deny
    Allow from all
</Directory>
(Don't know arcane details of above; but it works)
4- Tell the extension to web server by adding below.
 AddHandler cgi-script .cgi .pl
 (You can add other extensions too)

Problem- Source code is being displayed by web browser.
Solution-  You have not enabled below line (first step)
LoadModule cgid_module modules/mod_cgid.so

Problem- While accessing the URL, output has been sent to download instead of displaying properly by web browser.
Solution- Please make sure that vary first line to STDOUT is "print "Content-type: text/html\n\n"; "

Output format may vary as per requirement but take care of "\n\n".