Hadoop Installation:

You can configure hadoop in any Operating System, but the main power of hadoop comes with linux.

1. configure on windows -  using cigwin
            -  using virtual machine - install ubuntu. see below link.
(http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html)

2. configure on Linux machine.

/home/Admin/

su - root
password: root

adduser dav
passwd dav
(now my user name and password both are set as dav)

su - dav
password: dav
(now i am login with my user)

pwd

/home/dav/

step1: download hadoop - Use the latest stable release from Hadoop site and I used hadoop-1.2.1.tar.gz, it tarball file.
step2: Install java jdk1.6 or jdk1.7 or just put it here as it is.
step3: tar -zxvf hadoop-1.2.1.tar.gz
(now it)

mkdir hdfstmp
(it will create hdfstmp dir here)

go inside the hadoop directory and edit following files:
cd hadoop-1.2.1

vi conf/hadoop-env.sh
remove # tag from JAVA_HOME=/home/dav/jdk1.7.0_45
 
vi conf/core-site.xml 
 
<property>
  <name>hadoop.tmp.dir</name>
  <value>/data/hdfstmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
 
vi conf/hdfs-site.xml 
 
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property> 



vi conf/mapred-site.xml 


 


<property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>


 


vi slaves


enter slave machine ip address or write localhost ip.


 


vi masters


 enter master machine ip address or write localhost ip.


 


then format namenode


bin/hadoop  namenode -format




after successfuly formatting start hadoop


bin/start-all.sh


 


it will showing here 


jps


namenode


datanode


tasktracker


jobtracker


secondarynamenode


 


stop hadoop bin/stop-all.sh




jps


 


jps only show means hadoop is stopped.


































Comments











Post a Comment































Popular posts from this blog









Setup Nginx as a Reverse Proxy for Thingsboard running on different port/server





















  In this blog, we will Setup Nginx as a Reverse Proxy for Thingsboard IOT dashboard running on different port     Requirement:     Ubuntu Server   Thingsboard installed on Ubuntu server   Nginx installed on Ubuntu server       For installing please refer thingsboard documentation page     windows:    https://thingsboard.io/docs/user-guide/install/windows/     Linux:    https://thingsboard.io/docs/user-guide/install/linux/             Let us know in comment if you face any issue in installation we will help you.       NGINX installation:    sudo apt-get install nginx    sudo service nginx status   sudo service nginx start     Open your browser and you will see the default NGinx home page, nginx also runs on default port 80     In some case if you don't see default page up then check:      If using cloud machine then need to open "http port 80" on 0.0.0.0/0 or a specific ip  If still don't see UI then restart server, sometime it comes after restart.           Configure...








Read more










How to auto re-launch a YARN Application Master on a failure.





















  1)Use Case:     The fundamental idea of Hadoop2 (Map-Reduce + Yarn) is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager ( RM ) and per-application ApplicationMaster ( AM ). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.     The ResourceManager and per-node slave, the NodeManager ( NM ), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system.     The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.     The ResourceManager has two main components:     1. Scheduler: is responsible for allocating resources to the various running applications.     2. Appl...








Read more










Hive partitioned tables Issue with schema & PrestoDB





















 It's very strange for Hive and PrestoDB user that the schema of partitioned tables in Hive is defined on partition level as well. Partition level schema and table level schema in Apache Hadoop is letting complex.   Let's see the details in below example:   Table schema   In Hive you can change the schema of an existing table. Let’s say you have a table:   CREATE TABLE TEST1  (ID INT,   NAME STRING,   RATING INT  ) ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t" LINES TERMINATED BY '\n' STORED AS TEXTFILE;   We will focus on the third column named rating. We load few records for hdfs to this table. The file looks like this:   1         john          3.5  2          Dav          4.6  3          andy          5   hive -e "load data inpath 'input.txt' into table test1"    The third column has some decimal values, but we have defined this column as integer, so the we won’t see the decimal part in the data:   hive&gt; select * from te...








Read more