开发工具:
文件大小: 659kb
下载次数: 0
上传时间: 2019-08-03
详细说明：NULL
博文链接：https://57832638.iteye.com/blog/1983143Bulkload with custom Mapper
Reducer
Reducers
Mappers
(Text SortReducer) HFiles
(TsvImporter TextMapper)
One per
region
Bulkload in put files
o=s
≤
o∽o=-09
5<
Performance Test
Environment
3 Node cluster
Node 1-Name node, Data Node, Resource manager, Node manager, MAster,
Region server
Node 2- Zookeeper, Data Node, Region Server, Node manager
Node 3- Data Node, Region Server, Node manager
JVM
Default configurations
Performance Test
Test setup
500GB input data with 500 bytes in each line
Put to hdes with 64mb block size
Regions number of reducers -500
Number of mappers -8000
Column Family -1
Columns in ce-20
Test Report
Metric
Bulkload before changes
Bulkload after changes
(TsvImporter Mapper
(TsvImporterTextMapper
PutSort Reducer
TextSortReducer)
2000
14000
12000
500
10000
000
Reduce shuffle
8000
Data size in gB
6000
Bulkload before
500
4000
changes
Bytes read in gb
2000
0
Bulkload after
0
Bulkload Bulkload after
changes
Mapper Reducer
」ob
before
changes
Phase
Phase
Duration
hanges
Duration in duration in
Seconds Seconds
Conclusion
Noticed 55%improvement in throughput
> With improvement, Bytes shuffled are almost equal
to input data size where as with default ones it is 3
times more than the input data size
(系统自动生成,下载前可以参看下载内容)
下载文件列表
相关说明
- 本站资源为会员上传分享交流与学习,如有侵犯您的权益,请联系我们删除.
- 本站是交换下载平台，提供交流渠道，下载内容来自于网络，除下载问题外，其它问题请自行百度。
- 本站已设置防盗链，请勿用迅雷、QQ旋风等多线程下载软件下载资源，下载后用WinRAR最新版进行解压.
- 如果您发现内容无法下载，请稍后再次尝试；或者到消费记录里找到下载记录反馈给我们.
- 下载后发现下载的内容跟说明不相乎，请到消费记录里找到下载记录反馈给我们，经确认后退回积分.
- 如下载前有疑问，可以通过点击"提供者"的名字，查看对方的联系方式，联系对方咨询.
