Hadoop Tutorial | Spark Kafka Nosql and BigData tools for DWH

In this blog, we will Setup Nginx as a Reverse Proxy for Thingsboard IOT dashboard running on different port Requirement: Ubuntu Server Thingsboard installed on Ubuntu server Nginx installed on Ubuntu server For installing please refer thingsboard documentation page windows:  https://thingsboard.io/docs/user-guide/install/windows/ Linux:  https://thingsboard.io/docs/user-guide/install/linux/ Let us know in comment if you face any issue in installation we will help you. NGINX installation: sudo apt-get install nginx sudo service nginx status sudo service nginx start Open your browser and you will see the default NGinx home page, nginx also runs on default port 80 In some case if you don't see default page up then check: If using cloud machine then need to open "http port 80" on 0.0.0.0/0 or a specific ip If still don't see UI then restart server, sometime it comes after restart. Configure...

./sbin/start-master.sh you will get master is running on a hostname and a port number. copy that and put in blow command ./bin/spark-shell --driver-memory 4G --master spark://master-host:7077 --executor-memory 8G --executor-cores 1 --num-executors 3 import java.util.Properties val connectionProperties = new Properties() connectionProperties.put("user", "actualUsername") connectionProperties.put("password", "actualPassword") val jdbcUrl = "jdbc:mysql://hostname/dbname" val sqlquery = "(select * from t1 limit 10)tmp" val df = spark.read.jdbc(url=jdbcUrl, table=sqlquery, properties=connectionProperties) df.show for ms sql: https://docs.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url?view=sql-server-2017

                        Hbase installation on linux machine step by step Let's go for installtion of Hbase Nosql database on linux machine where hadoop cluster is already installed. step 1. Download hbase tar file hbase-0.94.2.tar.gz untar hbase-0.94.2.tar.gz. file tar -zxvf hbase-0.94.2.tar.gz. step 2. go inside hbase-0.94.2/conf/ and vi hbase-env.sh # The java implementation to use.  Java 1.6 required.  export JAVA_HOME=/home/dinesh1/jdk1.7.0_45 step 3. go inside hbase-0.94.2/conf/ and vi hbase-site.xml <configuration> <property> <!-- This is the location in HDFS that HBase will use to store its files --> <name>hbase.rootdir</name> <value>hdfs://192.168.5.134:54310/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> ...

Google Cloud Platform (GCP) is a set of physical assets, such as computers and hard disk drives and virtual resources like virtual machines, that are contained in Google's data centers around the globe with 99.99% up-time. Each data center location is in a global region. Each region is a collection of zones, which are isolated from each other within the region. By using cloude VM's one can use the resource to scale the requirement and can close the resource when don't need on the fly, which also minimize the cost. Very secure cloud and easy to use with less pricing. This distribution of resources provides several benefits, including redundancy in case of failure and reduced latency by locating resources closer to clients. This distribution also introduces some rules about how resources can be used together. About the GCP Services This overview introduces some of the commonly used Google Cloud Platform (GCP) services. . Types of services: Computing and hosting...

INSERT/Load Json data File in Cassandra (3.0+) table Requirement: Create Cassandra table where we can load json data into it. Here some of the column name is separated by space(like 'a b'). Load JSON file into the table. Challenge:  Cassandra support only 'CSV file' load in Table(As per my understanding and searched till now) by using 'COPY' command but not 'JSON' file. Resolution: As per the Cassandra document page, Cassandra supports CSV record level insert and file level insert both operation but for JSON it only support Record level insert by following command: cqlsh> INSERT INTO keyspace1.table1 JSON '{   "id" : "12",    "DB" : "Cassandra",    "ops" : "Insert",   "Project" : "Hadoop" }'; So If we want to insert whole file in the table then we need to loop for each object in JSON file and need to call insert query every tim...

It's very strange for Hive and PrestoDB user that the schema of partitioned tables in Hive is defined on partition level as well. Partition level schema and table level schema in Apache Hadoop is letting complex. Let's see the details in below example: Table schema In Hive you can change the schema of an existing table. Let’s say you have a table: CREATE TABLE TEST1 (ID INT,  NAME STRING,  RATING INT ) ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t" LINES TERMINATED BY '\n' STORED AS TEXTFILE; We will focus on the third column named rating. We load few records for hdfs to this table. The file looks like this: 1         john   3.5 2   Dav   4.6 3   andy   5 hive -e "load data inpath 'input.txt' into table test1" The third column has some decimal values, but we have defined this column as integer, so the we won’t see the decimal part in the data: hive&gt; select * from te...

Installation of Presto DB & Client Connection with Presto As we Already discussed about Presto DB that it is a distributed analytical query engine to run sql kind of query on data warehouse. So lets see the installation of Presto DB. Single node Presto DB Installation:  Here we will install Presto DB on single node Linux machine https://prestodb.io/docs/current/installation/deployment.html Multi node Presto DB Installation: Here we will install Presto DB on Three node Linux machine or the same can be install on existing Hadoop Cluster to run query on hive data. https://prestodb.io/docs/current/installation/deployment.html Client Connection with Presto: Presto DB client can be downloaded from Presto DB site: https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.161/presto-cli-0.161-executable.jar

Presto DB Power full Query Engine Presto DB is an open source distributed query engine to run interactive SQL(analytics query) on Big-Data which can be gigabytes to terabytes or petabytes.  Presto was designed for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook. Presto allows to querying data from Hadoop HDFS, Hive, Cassandra, relational databases or even proprietary data stores. A single presto query can combine data from multiple sources. The main goal of Presto to deliver analytics query result in sub-seconds to minutes on non-expensive hardware like hadoop cluster. It's fully free. Facebook uses Presto for interactive query against several internal data stores, including their 300 PB data warehouse. I personally tried presto DB on 3 node cluster with the data size of 1 TB to 3 TB data which resides on Hadoop HDFS and got the awesome performance in sub-seconds(calculations) ...

                             Add extra Hard Disk/Phisical memory at datanode As we are processing data on hadoop cluster we need to configure many things. The main thing is to manage temporary files which is generation in between map reduce. We can set map-reduce intermediate compression and map-reduce output compression. After setting every things we need atleat double/tripple space on datanode with respect to existing dataset on HDFS. So we can add some memory on hadoop cluster in 2 ways: 1. Add 1 or more datanode datanode 2. Add harddisk on the exising datanodes How to add extra HDD: First add a HardDisk on all datanode machine and mount on a point/name(example: /hdd2) Now create datanode directory in new harddisk (example: /hdd2/datanode) Now change ownership of /hdd2/datanode to hdfs/hadoop user then Stop one datanode and add the new Harddisk location in hdfs-site.xml <property> ...

Sqoop support only few file format(text, sequence, avro..etc), And If you want to store RDBMS data in HDFS in ORC(which is very compressed and fast file format as facebook said & used) you need to do this task in 2 steps. 1st import RDBMS data as text file and then Insert that data in ORC formatted table. (NOTE: We can do this using spark also). Here I am explaining how to do this using sqoop betch. I am using cdh5.4.0-hadoop-2.6, chd5.4.0-hive, apache-sqoop1.4.2 Hope you have all installed, you can do this with apache hadoop, hive also but some time it gives error because of version dependency. As per my knowledge If you can metch perfect hadoop& hive version then you'll not get any kind of error, other wise you have to face many error since apache foundation continuously improving every tools. If you are not sure then best to go with CHD, you can download tall file & install saperately. http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.4.0.tar.gz http://archiv...

1)Use Case: The fundamental idea of Hadoop2 (Map-Reduce + Yarn) is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager ( RM ) and per-application ApplicationMaster ( AM ). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. The ResourceManager and per-node slave, the NodeManager ( NM ), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks. The ResourceManager has two main components: 1. Scheduler: is responsible for allocating resources to the various running applications. 2. Appl...

Q: How can I run Linux shell script in every n minute/hours/monthly A: By using while loop in script or 'CRON'     'CRON' is best Choice Cron is a daemon found on most Unix/linux systems that runs scheduled commands at the specified intervals. You add a script to the list by copying it to the folder of your choice: cron.daily cron.hourly cron.monthly cron.weekly These folders are typically found in /etc OR Just type below command on consol and editor will open, $crontab -e In this there ia a line like # * * * * * command Remove # from the begining of The line and set like you want for 10 minute */10 * * * * ./script for 2 hours 0 */2 * * * here ./script is may be linux command or a script. By using this we can RUN Hadoop/Hive/PIG scripts on every Interval which we want. =================xxxxxxxxxxxxxxxx============================ My Scripts ============================================================= script.sh file in Home d...

                    MongoDB Replication Configuration: while replication in mongodb if it show error like "not master" then run rs.slaveOk() command on secondary node side it will solve issue. ==================================================== Replication- mongoDB 1.Start by creating a data directory for each replica set member: mkdir /data/node1 mkdir /data/node2 mkdir /data/arbiter 2. mongod --replSet myapp --dbpath /data/node1 --port 40000 mongod --replSet myapp --dbpath /data/node2 --port 40001 mongod --replSet myapp --dbpath /data/arbiter --port 40002 3. run mongo hostname:40000 to rum client on primary, and then run the rs.initiate() command: > rs.initiate() { "info2" : "no configuration explicitly specified -- making one", "me" : "arete:40000", "info" : "Config now saved locally. Should come online in about a minute .", "ok" : 1 } 4.You can now add the o...

                   MongoDB - Document Oriented NoSQL DataBase: MongoDB is one of several database types to arise in the mid-2000s under the NoSQL banner. Instead of using tables and rows as in relational databases , MongoDB is built on an architecture of collections and documents. Documents comprise sets of key-value pairs and are the basic unit of data in MongoDB. Collections contain sets of documents and function as the equivalent of relational database tables. Like other NoSQL databases, MongoDB supports dynamic schema design, allowing the documents in a collection to have different fields and structures. The database uses a document storage and data interchange format called BSON, which provides a binary representation of JSON -like documents. Automatic sharding enables data in a collection to be distributed across multiple systems for horizontal scalability as data volumes ...

                          100's of blogs Related to BigData: http://blogs.the451group.com/opensource/ href="http://abeautifulwww.com" href="http://timmanns.blogspot.com/" href="http://www.behind-the-enemy-lines.com/" href="http://www.acthomas.ca/" href="http://abbottanalytics.blogspot.com/" href="http://www.advancednflstats.com" href="http://patilv.github.io/" href="http://blog.smola.org/" href="http://blog.markus-breitenbach.com/" href="http://allthingsdistributed.com" href="http://aws.typepad.com/aws" href="http://www.analyticbridge.com/profiles/blog/list" href="http://www.analyticsvidhya.com/blog/" href="http://www.applieddatalabs.com" href="http://atbrox.com" href="http://www.bitquill.net/blog" href="http://radar.oreilly.com/ben/" href="http://benfry.com/writing/" href="htt...

Process xml file on hadoop OR Load xml file data in Hive/HBase: My program code: Here I have XmlDriver class, in that i have xmlRecodReader class, map class & reduce class and it will process xml data and generate comma seprated columns in hdfs. Raw data file like: <current_observation version=\"1.0"....>   <latitude>1</latitude>   <longitude>2</longitude>   <pressure_mb>3</pressure_mb>   <wind_mph>4</wind_mph>   <wind_dir>5</wind_dir>   <windchill_c>6</windchill_c>   <temp_c>7</temp_c>   <wind_degrees>8</wind_degrees> </current_observation>  import java.io.ByteArrayInputStream; import java.io.IOException; import javax.xml.stream.XMLInputFactory; import javax.xml.stream.XMLStreamConstants; import javax.xml.stream.XMLStreamReader; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputS...

updating csv document in solr [user@localhost exampledocs]$ curl http://localhost:8983/solr/update/csv --data- binary @books.csv -H 'Content-type:text/plain; charset=utf-8' after above command you will see like this: <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"><int name="status">0</int><int name="QTime">60</int></lst> </response> [user@localhost exampledocs]$ curl http://localhost:8983/solr/update/csv --data-binary @books.csv -H 'Content-type:text/plain; charset=utf-8' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"><int name="status">0</int><int name="QTime">61</int></lst> </response> [user@localhost exampledocs]$ means data is updating.. i am also working on this so i'll change/upd...

   Solr Installation and New core Configuration: Apache Solr achieve fast search responses because, instead of searching the text directly, it searches an index. This is like retrieving pages in a book based on a word by scanning the index at the back of a book, as opposed to searching every word of every page of the book. This type of index is called an inverted index, because it inverts a page-centric data structure (page->words) to a keyword-centric data structure (word->pages). Solr stores this index in a directory called index in the data directory. Apache Solr is powered by Lucene, a powerful open-source full-text search library, under the hood. The relationship between Solr and Lucene, is like that of the relationship between a car and engine. In Solr, a Document is the unit of search and index. An index consists of one or more Documents, and a Document consists of one or more Fields. Solr Installation and New core Configuration: 1. Installation- Do...

            For Hadoop Interview Question See: Hadoop Interview Questions and Answers by rohit kapa from kapa rohit

                            Sqoop Installation: Sqoop is usefull for import/export sql or rdbms data files or table in hadoop HDFS or in Nosql. Download Sqoop 1.4.2 $ wget http://www.eu.apache.org/dist/sqoop/1.4.2/sqoop- 1.4.2.bin__hadoop-1.0.0.tar.gz Extract Sqoop 1.4.2 tarball, $ tar -zxvf sqoop-1.4.2.bin__hadoop-1.0.0.tar.gz Update the hadooprc.sh with export SQOOP_HOME $ cat hadooprc.sh export SQOOP_HOME=/data/sqoop-1.4.2 export JAVA_HOME=/usr/java/jdk1.7.0_05 export HADOOP_HOME=/data/hadoop-1.0.1 export HBASE_HOME=/data/ahbase-0.94.3 export PATH=$PIG_HOME/bin:$SQOOP_HOME/bin:$HIVE_HOME/bin:$JAVA_HOME/bin:$HA DOOP_HOME/bin:$PATH:$HBASE_HOME/bin export CLASSPATH=$JAVA_HOME:/data/hadoop-1.0.1/hadoop-core- 1.0.1.jar:$PIG_HOME/pig-0.10.0.jar Download the Mysql connector jar from, http://dev.mysql.com/downloads/connector/j/ extract ...