What is Big Data
 
*  Every day, world create 2.5 quintillion bytes of data so much that 90% of the data in the world today has been created in the last two years alone.

* Gartner defines Big Data as high volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.

*According to IBM, 80% of data captured today is unstructured, from sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, to name a few. All of this unstructured data is also Big Data.


* Huge Competition in the Market:
                 - Retails – Customer analytics
                 - Travel – travel pattern of the customer
                 - Website – Understand users navigation pattern, interest, conversion, etc
                 - Sensors, satellite, geospatial Data
                 - Military and intelligence
 
1.  Volume
      - Today we are living in the world of data. There are multiple factors contributing in data growth
      - Huge volumes of data are generated from various sources:
      - Transaction based data (stored through years)
      - Text, Images, Videos from Social Media
      - Increased amounts of data generated by sensors
      -  Turn 12 terabytes of Tweets created each day into improved product sentiment analysis
      -  Convert 350 billion annual meter readings to better predict power consumption
      -  Turn billions of customer complaints to analyze root cause of customer churn

2.  Velocity 
       -  According to Gartner, velocity "means both how fast data is being produced and how fast the data must be
processed to meet demand." 
       -  Scrutinize 5 million trade events created each day to identify potential fraud 
       -  Analyze customer’s searching/buying pattern and show them advertisement of attractive offers in real time
       -  Take Google’s example, about processing of the data:
       -  As soon as a blog is posted it comes into the search result.
       - Even ads in the mail are highly content driven
 
3.  Variety
       -  Data today comes in all types of formats – from traditional databases to hierarchical data stores created by end users and OLAP systems, to text documents, email, meter-collected data, video, audio, stock ticker data and financial transactions.
 ============*****==============

Comments

Post a Comment

Popular posts from this blog

Setup Nginx as a Reverse Proxy for Thingsboard running on different port/server

In this blog, we will Setup Nginx as a Reverse Proxy for Thingsboard IOT dashboard running on different port Requirement: Ubuntu Server Thingsboard installed on Ubuntu server Nginx installed on Ubuntu server For installing please refer thingsboard documentation page windows:  https://thingsboard.io/docs/user-guide/install/windows/ Linux:  https://thingsboard.io/docs/user-guide/install/linux/ Let us know in comment if you face any issue in installation we will help you. NGINX installation: sudo apt-get install nginx sudo service nginx status sudo service nginx start Open your browser and you will see the default NGinx home page, nginx also runs on default port 80 In some case if you don't see default page up then check: If using cloud machine then need to open "http port 80" on 0.0.0.0/0 or a specific ip If still don't see UI then restart server, sometime it comes after restart. Configure
Read more

How to auto re-launch a YARN Application Master on a failure.

1)Use Case: The fundamental idea of Hadoop2 (Map-Reduce + Yarn) is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager ( RM ) and per-application ApplicationMaster ( AM ). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. The ResourceManager and per-node slave, the NodeManager ( NM ), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks. The ResourceManager has two main components: 1. Scheduler: is responsible for allocating resources to the various running applications. 2. Appl
Read more

Read JSON File in Cassandra

Image
INSERT/Load Json data File in Cassandra (3.0+) table Requirement: Create Cassandra table where we can load json data into it. Here some of the column name is separated by space(like 'a b'). Load JSON file into the table. Challenge:  Cassandra support only 'CSV file' load in Table(As per my understanding and searched till now) by using 'COPY' command but not 'JSON' file. Resolution: As per the Cassandra document page, Cassandra supports CSV record level insert and file level insert both operation but for JSON it only support Record level insert by following command: cqlsh> INSERT INTO keyspace1.table1 JSON '{   "id" : "12",    "DB" : "Cassandra",    "ops" : "Insert",   "Project" : "Hadoop" }'; So If we want to insert whole file in the table then we need to loop for each object in JSON file and need to call insert query every tim
Read more