Hadoop Tutorial | Spark Kafka Nosql and BigData tools for DWH

./sbin/start-master.sh you will get master is running on a hostname and a port number. copy that and put in blow command ./bin/spark-shell --driver-memory 4G --master spark://master-host:7077 --executor-memory 8G --executor-cores 1 --num-executors 3 import java.util.Properties val connectionProperties = new Properties() connectionProperties.put("user", "actualUsername") connectionProperties.put("password", "actualPassword") val jdbcUrl = "jdbc:mysql://hostname/dbname" val sqlquery = "(select * from t1 limit 10)tmp" val df = spark.read.jdbc(url=jdbcUrl, table=sqlquery, properties=connectionProperties) df.show for ms sql: https://docs.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url?view=sql-server-2017

                        Hbase installation on linux machine step by step Let's go for installtion of Hbase Nosql database on linux machine where hadoop cluster is already installed. step 1. Download hbase tar file hbase-0.94.2.tar.gz untar hbase-0.94.2.tar.gz. file tar -zxvf hbase-0.94.2.tar.gz. step 2. go inside hbase-0.94.2/conf/ and vi hbase-env.sh # The java implementation to use.  Java 1.6 required.  export JAVA_HOME=/home/dinesh1/jdk1.7.0_45 step 3. go inside hbase-0.94.2/conf/ and vi hbase-site.xml <configuration> <property> <!-- This is the location in HDFS that HBase will use to store its files --> <name>hbase.rootdir</name> <value>hdfs://192.168.5.134:54310/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>192.168.5.134</value

Google Cloud Platform (GCP) is a set of physical assets, such as computers and hard disk drives and virtual resources like virtual machines, that are contained in Google's data centers around the globe with 99.99% up-time. Each data center location is in a global region. Each region is a collection of zones, which are isolated from each other within the region. By using cloude VM's one can use the resource to scale the requirement and can close the resource when don't need on the fly, which also minimize the cost. Very secure cloud and easy to use with less pricing. This distribution of resources provides several benefits, including redundancy in case of failure and reduced latency by locating resources closer to clients. This distribution also introduces some rules about how resources can be used together. About the GCP Services This overview introduces some of the commonly used Google Cloud Platform (GCP) services. . Types of services: Computing and hosting

INSERT/Load Json data File in Cassandra (3.0+) table Requirement: Create Cassandra table where we can load json data into it. Here some of the column name is separated by space(like 'a b'). Load JSON file into the table. Challenge:  Cassandra support only 'CSV file' load in Table(As per my understanding and searched till now) by using 'COPY' command but not 'JSON' file. Resolution: As per the Cassandra document page, Cassandra supports CSV record level insert and file level insert both operation but for JSON it only support Record level insert by following command: cqlsh> INSERT INTO keyspace1.table1 JSON '{   "id" : "12",    "DB" : "Cassandra",    "ops" : "Insert",   "Project" : "Hadoop" }'; So If we want to insert whole file in the table then we need to loop for each object in JSON file and need to call insert query every tim

It's very strange for Hive and PrestoDB user that the schema of partitioned tables in Hive is defined on partition level as well. Partition level schema and table level schema in Apache Hadoop is letting complex. Let's see the details in below example: Table schema In Hive you can change the schema of an existing table. Let’s say you have a table: CREATE TABLE TEST1 (ID INT,  NAME STRING,  RATING INT ) ROW FORMAT DELIMITED FIELDS TERMINATED BY "\t" LINES TERMINATED BY '\n' STORED AS TEXTFILE; We will focus on the third column named rating. We load few records for hdfs to this table. The file looks like this: 1         john   3.5 2   Dav   4.6 3   andy   5 hive -e "load data inpath 'input.txt' into table test1" The third column has some decimal values, but we have defined this column as integer, so the we won’t see the decimal part in the data: hive&gt; select * from test1; OK

Installation of Presto DB & Client Connection with Presto As we Already discussed about Presto DB that it is a distributed analytical query engine to run sql kind of query on data warehouse. So lets see the installation of Presto DB. Single node Presto DB Installation:  Here we will install Presto DB on single node Linux machine https://prestodb.io/docs/current/installation/deployment.html Multi node Presto DB Installation: Here we will install Presto DB on Three node Linux machine or the same can be install on existing Hadoop Cluster to run query on hive data. https://prestodb.io/docs/current/installation/deployment.html Client Connection with Presto: Presto DB client can be downloaded from Presto DB site: https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.161/presto-cli-0.161-executable.jar