High-performance Persistent Storage System for Bigdata Analysis: Performance Evaluation of Hdd and Ssd on 10gige, Ipoib & Rdma-ib with Hadoop Cluster Performance Benchmarking System

Hadoop & Map reduce today are facing huge amounts of data and are moving towards ubiquitous for big data storage and processing. This has made it an essential feature to evaluate and characterize the Hadoop file system & its deployment through extensive benchmarking. HiBench is an essential part of Hadoop and is comprehensive benchmark suit that consist of a complete deposit of Hadoop applications having micro bench marks & real time applications for the purpose of bench marking the performance of Hadoop on the available type of storage device and machine configuration. This is helpful to optimize the performance & improve the support towards the limitations of Hadoop system. In this research work analysis & characterization of the performance of external sorting algorithm in Hadoop (MapReduce) with SSD & HDD that are connected with various Interconnect technologies like 10GigE, IPoIB & RDBA-IB. In addition, the traditional servers & old Cloud systems can be upgraded by hardware up gradations to perform at par with the modern technologies to handle loads with the use of Modern storage devices and interconnect networking systems that provide improved throughput and lower latency.