Store files in a tiered storage, WAL is stored in RAMDISK, some smaller files (not larger than 1.5GB) are stored in SSD, and all the other files are stored in HDD including all archived files. Store files in a tiered storage, WAL and flushed HFiles are stored in SSD, and all the other files are stored in HDD including all archived files and compacted files. Store files in a tiered storage, WAL is stored in SSD, some smaller files (not larger than 1.5GB) are stored in SSD, and all the other files are stored in HDD including all archived files. Store files in a tiered storage, WAL is stored in RAMDISK, and all the other files are stored in HDD. different storage mixed together in one test), WAL is stored in RAMDISK, and all the other files are stored in SSD. Compare the performance by 1TB data with cases in tiered storage. Compare the performance by 50GB data with the 1st case. We have to limit data size to 50GB in this case due to the capacity limitation of RAMDISK. We split the table into 210 regions in 1 TB dataset cases to avoid region split at runtime, and we pre-split the table into 18 regions in 50 GB dataset cases. YCSB (Yahoo! Cloud Serving Benchmark, a widely used open source framework for evaluating the performance of data-serving systems) is used as the test workload.Įleven test cases are evaluated in this study. We test the write performance in HBase with a tiered storage in HDFS and compare the performance when storing different HBase data into different storages. Files stored into RAMDISK will only be cached in memory. Note: RAMDISK is logical device emulated with memory. And we believe our results hold for PCI-e SSD and other fast storage types. Due to hardware unavailability, we have to use RAMDISK to perform this emulation. RAMDISK is used to emulate the extremely high performance PCI-e NVMe based SSDs and coming faster storage (e.g. HDD is the most popular storage in current usages, SATA SSD is a faster storage which is more and more popular now. Three different types of storage (HDD, SSD and RAMDISK) are evaluated. We leverage the hierarchy storage management support in HDFS to store different categories of HBase data on different media. In this study, we study the HBase write performance on different storage media.
#Hbase storage policy disk archive how to
How to predict what kind of performance a cluster can have with different storage combinations? How to achieve the best balance between performance and cost?
What is the maximum performance user can achieve by using fast storage? In general, this guide considers the following questions:
This study provides a guide for cluster setup with different storage media. However, these fast storage types are still expensive and are capacity limited.
#Hbase storage policy disk archive software
This is part 1 of a 7 part report by HBase Contributor, Jingcheng Du and HDFS contributor, Wei Zhou (Jingcheng and Wei are both Software Engineers at Intel)Īs more and more fast storage types (SSD, NVMe SSD, etc.) emerge, a methodology is necessary for better throughput and latency when using big data. HDFS HSM and HBase: Introduction (Part 1 of 7)
0 kommentar(er)
