NO SQL

• combination of two words: No and SQL
• Technology/product that counters SQL
• No RDBMS or No relational
• “Not Only SQL.”
• NoSQL is not a single product or even a single technology
• It represents a class of products and a collection of diverse, and sometimes related, concepts about data storage and manipulation

Bit of History

• (RDBMS) has its own set of problems when applied to massive amounts of data
• Problems related to efficient processing, effective parallelization, scalability, and costs
• Denormalizing tables, dropping constraints can help an RDBMS scale, but after these modifications an RDBMS starts resembling a NoSQL product

• NoSQL alleviates the problems that RDBMS imposes and makes it easy to work with large sparse data, but in turn takes away the power of transactional integrity and flexible indexing and querying.
• Ironically, one of the features most missed in NoSQL is SQL, and product vendors in the space are making all sorts of attempts to bridge this gap

• Google has search engine and other applications, including Google Maps, Google Earth, GMail, Google Finance, and Google Apps.
• Google’s approach was to solve the problem at every level of the application stack.
• The goal was to build a scalable infrastructure for parallel processing of large amounts of data.

• Google then created a full mechanism that included a distributed filesystem, a column-family-oriented data store, a distributed coordination system, and a MapReduce-based parallel algorithm execution environment.
• Google published and presented a series of papers explaining some of the key pieces of its infrastructure.

​

​

​

​

• Google’s papers to the public spurred a lot of interest among open-source developers.
• The creators of the open-source search engine, Lucene, were the first to develop an open-source version that replicated some of the features of Google’s infrastructure.

• the core Lucene developers joined Yahoo, where with the help of a host of other contributors, they created a parallel universe that mimicked all the pieces of the Google distributed computing stack.
• it’s important to note that the emergence of Hadoop laid the groundwork for the rapid growth of NoSQL.

​

​

• A year after the Google papers had catalyzed interest in parallel scalable processing and nonrelational distributed data stores, Amazon decided to share some of its own success story.
• In 2007, Amazon presented its ideas of a distributed highly available and eventually consistent data store named Dynamo.

• web giants — Google and Amazon — several new products emerged in this space.
• In less than 5 years, NoSQL and related concepts for managing big data have become widespread.
• use cases have emerged from many well-known companies, including Facebook, Netflix, Yahoo, EBay, Hulu, IBM, and many more. Many of these companies have also contributed by open sourcing their extensions and newer products to the world.

Big Data

• The movie Avatar took up 1 petabyte of storage space for the rendering of 3D CGI effects

Scalability

• ability of a system to increase throughput with addition of resources to address the extra load
• Additional large and powerful resource to meet the additional demands
• Use cluster of ordinary machines to work as a unit.
• The involvement of large, powerful machines is typically classified as vertical scalability.
• vertical scaling ---- Providing super computers with many CPU cores and large amounts of storage is a typical vertical scaling solution.
• are typically expensive and proprietary.

​

horizontal scalability.

• The alternative to vertical scalability
• Horizontal scalability involves a cluster of commodity systems where the cluster scales as load increases.
• Horizontal scalability typically involves adding additional nodes to serve additional load.
• big data and the need for large-scale parallel processing requires horizontally scalable infrastructures.
• Google, Amazon, Facebook, eBay, and Yahoo! involve a very large number of servers.
• Some of these infrastructures have thousands and even hundreds of thousands of servers.

TYPES of NO SQL

1. Document databases: These databases store data as semi-structured documents, such as JSON or XML, and can be queried using document-oriented query languages.

Document-based: Examples – MongoDB, CouchDB, Cloudant

​

​

​

DOCUMENT DATABASES

• typically JSON (JavaScript Object Notation), and not documents or spreadsheets (though these could be stored too)
• Document databases treat a document as a whole and avoid splitting a document into its constituent name/value pairs.
• At a collection level, this allows for putting together a diverse set of documents into a single collection.

• Document databases allow indexing of documents on the basis of not only its primary identifier but also its properties.
• A few different open-source document databases are available today but the most prominent among the available options are MongoDB and CouchDB.

• example

Key-value stores: 

These databases store data as key-value pairs, and are optimized for simple and fast read/write operations.

Key/ value store: Examples –Redis, Coherence

​

KEY/VALUE STORES

• A HashMap or an associative array is the simplest data structure that can hold a set of key/value pairs.
• Such data structures are extremely popular because they provide a very efficient, big O(1) average algorithm running time for accessing data.
• Key/value pairs are of varied types:
• some keep the data in memory and some store data to disk.

​

• Key/value pairs can be distributed and held in a cluster of nodes.
• A simple, yet powerful, key/value store is Oracle’s Berkeley DB. Berkeley DB is a pure storage engine where both key and value are an array of bytes.

• Another type of key/value store in common use is a cache. A cache provides an in-memory snapshot of the most-used data in an application.
• The purpose of cache is to reduce disk I/O.

​

SORTED ORDERED COLUMN-ORIENTED STORES

• These databases store data as column families, which are sets of columns that are treated as a single entity. They are optimized for fast and efficient querying of large amounts of data.

Column: Examples – Hbase, Big Table, Accumulo

​

​

​

SORTED ORDERED COLUMN-ORIENTED STORES

• allows data to be stored effectively.
• It avoids consuming space when storing nulls by simply not storing a column when a value doesn’t exist for that column.
• Each unit of data can be thought of as a set of key/value pairs

​

• unit itself is identified with the help of a primary identifer, often referred to as the primary key.
• Bigtable and its clones tend to call this primary key the row-key.
• units are stored in an ordered-sorted manner. The units of data are sorted and ordered on the basis of the row-key.

​

GRAPH DATABASES

These databases store data as nodes and edges, and are designed to handle complex relationships between data.

Graph Databases: Examples – Amazon Neptune, Neo4j

​

GRAPH DATABASES

• Neo4j and FlockDB: Neo4J is an ACID-compliant graph database. It facilitates rapid traversal of graphs.

Column : HBASE

• Official Online Resources — http://hbase.apache.org.
• History — Created at Powerset (now part of Microsoft) in 2007. Donated to the Apache foundation before Powerset was acquired by Microsoft.
• Technologies and Language — Implemented in Java.
• Access Methods — A JRuby shell allows command-line access to the store.
• . A Java API is available with the distribution.

• Query Language — No native querying language. Hive (http://hive.apache.org) provides a SQL-like interface for HBase.
• Open-Source License — Apache License version 2.
• Who Uses It — Facebook, StumbleUpon, Hulu, Ning, Mahalo, Yahoo!, and others.

WHAT IS THRIFT?

• thrift is a software framework and an interface defi nition language
• that allows cross-language services and API development.
• Services generated using Thrift work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml.
• Thrift was created by Facebook in 2007.
• It’s an Apache incubator project.

Hypertable

• History — Created at Zvents in 2007. Now an independent open-source project.
• Technologies and Language — Implemented in C++, uses Google RE2 regular expression library. RE2 provides a fast and efficient implementation. Hypertable ----performance boost over HBase, reduce time and cost when dealing with large amounts of data.
• Access Methods — A command-line shell is available. In addition, a Thrift interface is supported. Language bindings have been created based on the Thrift interface. A creative.

• Query Language — HQL (Hypertable Query Language) is a SQL-like abstraction for querying Hypertable data. Hypertable also has an adapter for Hive.
• Open-Source License — GNU GPL version 2.
• Who Uses It — Zvents, Baidu (China’s biggest search engine), Rediff (India’s biggest portal).

Column-Cloudata

KEY-VALUE

Key Value- Cassandra

Document

Graph