Hadoop Installation:

You can configure hadoop in any Operating System, but the main power of hadoop comes with linux.

1. configure on windows -  using cigwin
            -  using virtual machine - install ubuntu. see below link.
(http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/SingleNodeSetup.html)

2. configure on Linux machine.

/home/Admin/

su - root
password: root

adduser dav
passwd dav
(now my user name and password both are set as dav)

su - dav
password: dav
(now i am login with my user)

pwd

/home/dav/

step1: download hadoop - Use the latest stable release from Hadoop site and I used hadoop-1.2.1.tar.gz, it tarball file.
step2: Install java jdk1.6 or jdk1.7 or just put it here as it is.
step3: tar -zxvf hadoop-1.2.1.tar.gz
(now it)

mkdir hdfstmp
(it will create hdfstmp dir here)

go inside the hadoop directory and edit following files:
cd hadoop-1.2.1

vi conf/hadoop-env.sh
remove # tag from JAVA_HOME=/home/dav/jdk1.7.0_45
 
vi conf/core-site.xml 
 
<property>
  <name>hadoop.tmp.dir</name>
  <value>/data/hdfstmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
 
vi conf/hdfs-site.xml 
 
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property> 



vi conf/mapred-site.xml 


 


<property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>


 


vi slaves


enter slave machine ip address or write localhost ip.


 


vi masters


 enter master machine ip address or write localhost ip.


 


then format namenode


bin/hadoop  namenode -format




after successfuly formatting start hadoop


bin/start-all.sh


 


it will showing here 


jps


namenode


datanode


tasktracker


jobtracker


secondarynamenode


 


stop hadoop bin/stop-all.sh




jps


 


jps only show means hadoop is stopped.


































Comments











Post a Comment































Popular posts from this blog









Setup Nginx as a Reverse Proxy for Thingsboard running on different port/server





















  In this blog, we will Setup Nginx as a Reverse Proxy for Thingsboard IOT dashboard running on different port     Requirement:     Ubuntu Server   Thingsboard installed on Ubuntu server   Nginx installed on Ubuntu server       For installing please refer thingsboard documentation page     windows:    https://thingsboard.io/docs/user-guide/install/windows/     Linux:    https://thingsboard.io/docs/user-guide/install/linux/             Let us know in comment if you face any issue in installation we will help you.       NGINX installation:    sudo apt-get install nginx    sudo service nginx status   sudo service nginx start     Open your browser and you will see the default NGinx home page, nginx also runs on default port 80     In some case if you don't see default page up then check:      If using cloud machine then need to open "http port 80" on 0.0.0.0/0 or a specific ip  If still don't see UI then restart server, sometime it comes after restart.           Configure








Read more










How to auto re-launch a YARN Application Master on a failure.





















  1)Use Case:     The fundamental idea of Hadoop2 (Map-Reduce + Yarn) is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager ( RM ) and per-application ApplicationMaster ( AM ). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.     The ResourceManager and per-node slave, the NodeManager ( NM ), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system.     The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.     The ResourceManager has two main components:     1. Scheduler: is responsible for allocating resources to the various running applications.     2. Appl








Read more










Read JSON File in Cassandra




















Image







   INSERT/Load Json data File in Cassandra (3.0+) table      Requirement:   Create Cassandra table where we can load json data into it. Here some of the column name is separated by space(like 'a b').  Load JSON file into the table.       Challenge:    Cassandra support only 'CSV file' load in Table(As per my understanding and searched till now) by using 'COPY' command but not 'JSON' file.     Resolution:   As per the Cassandra document page, Cassandra supports CSV record level insert and file level insert both operation but for JSON it only support Record level insert by following command:   cqlsh> INSERT INTO keyspace1.table1 JSON '{    "id" : "12",     "DB" : "Cassandra",     "ops" : "Insert",    "Project" : "Hadoop" }';        So If we want to insert whole file in the table then we need to loop for each object in JSON file and need to call insert query every tim








Read more