How to install apache sqoop

                            Sqoop Installation:

Sqoop is usefull for import/export sql or rdbms data files or table in hadoop HDFS or in Nosql.

Download Sqoop 1.4.2
$ wget http://www.eu.apache.org/dist/sqoop/1.4.2/sqoop-
1.4.2.bin__hadoop-1.0.0.tar.gz
Extract Sqoop 1.4.2 tarball,
$ tar -zxvf sqoop-1.4.2.bin__hadoop-1.0.0.tar.gz
Update the hadooprc.sh with export SQOOP_HOME
$ cat hadooprc.sh
export SQOOP_HOME=/data/sqoop-1.4.2
export JAVA_HOME=/usr/java/jdk1.7.0_05
export HADOOP_HOME=/data/hadoop-1.0.1
export HBASE_HOME=/data/ahbase-0.94.3
export
PATH=$PIG_HOME/bin:$SQOOP_HOME/bin:$HIVE_HOME/bin:$JAVA_HOME/bin:$HA
DOOP_HOME/bin:$PATH:$HBASE_HOME/bin
export CLASSPATH=$JAVA_HOME:/data/hadoop-1.0.1/hadoop-core-
1.0.1.jar:$PIG_HOME/pig-0.10.0.jar
Download the Mysql connector jar from,
http://dev.mysql.com/downloads/connector/j/
extract it, copy the mysql-connector-java-5.1.22-bin.jar into
SQOOP_HOME/lib
Install Mysql-Server
$sudo yum install mysql-server
$sudo service mysqld status
To Start Mysql-server,
$sudo service mysqld start
To test connectivity with MySQL, List Databases from Sqoop
$ sqoop-list-databases --connect jdbc:mysql://localhost/ --username
root --password ""
$ sqoop list-tables --connect jdbc:mysql://localhost/test --username
root --password ""

Create a Mysql Table, and insert records.
$ mysql –u root –p
mysql> create database bigdatatraining_in;
mysql> use bigdatatraining_in;
mysql> create table employee (
id INT AUTO_INCREMENT PRIMARY KEY,
name varchar(20),
dept varchar(10),
salary int(10)
);
mysql> insert into employee values(100,'Thomas','Sales',5000);
mysql> insert into employee values(200,'Jason','Technology',5500);
mysql> insert into employee values(300,'Mayla','Technology',7000);
mysql> insert into employee values(400,'Nisha','Marketing',9500);
mysql> insert into employee values(500,'Randy','Technology',6000);

Set up a user account to connect via Sqoop. Grant permissions to the user to access the database over the network: (1.) Log into MySQL as root mysql -u root -p<ThisIsMyPassword>. (2.) Issue the following command:
mysql> grant all privileges on test.* to 'testuser'@'%' identified by 'testpassword'

If the table name is employee, under database test, the following command will import
the mysql table into hdfs.
sqoop import --connect jdbc:mysql://localhost/bigdatatraining_in --
username root --password "" --table employee -m1
hadoop fs -ls
hadoop fs -ls /user/hadoop/employee
hadoop fs -cat /user/hadoop/employee/part-m-00000
We should see contents of the Mysql Table.

........................................................................................................................................
[user@localhost sqoop-1.4.3.bin__hadoop-1.0.0]$ bin/sqoop-list-databases --

[user@localhost sqoop-1.4.3.bin__hadoop-1.0.0]$ bin/sqoop-list-databases --connect jdbc:mysql://192.168.5.59/ --username root --password ""

[user@localhost sqoop-1.4.3.bin__hadoop-1.0.0]$ bin/sqoop-list-tables --connect jdbc:mysql://192.168.5.59/newdb --username root --password ""

[user@localhost sqoop-1.4.3.bin__hadoop-1.0.0]$ bin/sqoop-import --connect jdbc:mysql://192.168.5.59/newdb --username root --password "" --m 1 --table employee

[user@localhost hadoop-1.2.1]$ bin/hadoop fs -cat /user/noah/employee/part-m-00000

Comments

  1. This comment has been removed by a blog administrator.

    ReplyDelete

Post a Comment

Popular posts from this blog

Setup Nginx as a Reverse Proxy for Thingsboard running on different port/server

How to auto re-launch a YARN Application Master on a failure.

Read JSON File in Cassandra