js333 > 计算机互联网 > 安装记录,分布式安装

原标题:安装记录,分布式安装

浏览次数:87 时间:2019-11-05

HBase 0.94.21+ZooKeeper-3.4.6 分布式安装

Hadoop 1.0.0的安装这里就不讲了,请参照 http://www.linuxidc.com/Linux/2012-02/54950.htm ,里面已经相信介绍了.

HBase的几种调优(GC策略,flush,compact,split)

 

一:GC的调优

1.jvm的内存

  新生代:存活时间较短,一般存储刚生成的一些对象
  老年代:存活时间较长,主要存储在应用程序中生命周期较长的对象
  永久代:一般存储meta和class的信息

 

2.GC策略

  Parrallel New Collector,垃圾回收策略

  并行标记回收器(Concurrent Mark-Sweep Collector)

 

3.Parrallel New Collector

  速度快,但是数据量一大,容易造成GC停顿,一般应用在新生代

 

4.Concurrent Mark-Sweep Collector

速度相对来说慢一点,但是对于大数据量来说,避免GC停顿

 

5.设置(在hbase-env.sh中)

export   HBASE_REGIONSERVER_OPTS=”-Xmx8g -Xms8G        -Xmn128m -XX:UseParNewGC     -XX:UseConcMarkSweepGC -    XX:CMSInitiatingOccupancyFraction=70            -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps    -Xloggc:$HBASE_HOME/logs/gc-${hostname}-hbase.log”

 

二:flush

1.概述 

  写的过程:先往WAL写,往memstore写,memstore达到阈值,开始溢写到HDFS
  memstore配置:默认达到128M开始溢写,自动flush
    <property>
      <name>hbase.hregion.memstore.flush.size</name>
      <value>134217728</value>
    </property>

2.新建表s3

 

3.添加数据

  金沙js333娱乐场 1

4.在hdfs上的效果

  无

 

5.flush

  金沙js333娱乐场 2

 

6.在hdfs上出现

  金沙js333娱乐场 3

7.新添加数据

  金沙js333娱乐场 4

8.在hdfs上将会出现两个storefile

  金沙js333娱乐场 5

三:compact

1.概述

  溢写以后生成许多storefile
  minor compaction:最早生成的几个storefile会被合并,这一步不会对标记的删除和过期的数据进行处理,经常发生,数据量较小,占用资源不多
  major compaction:默认值是7天执行一次
  设置  

    <property>
      <name>hbase.hregion.majorcompaction</name>
      <value>604800000</value>
    </property>

2.危害
  -》消耗大量的资源,对hbase的性能产生影响
  -》会导致hbase的应用阻塞
  企业中一般关闭自动触发,使用手动触发 major_compact

 

3.操作

  在自动触发之前手动触发

  金沙js333娱乐场 6

 

4.在hdfs上的效果

  合并成一个文件

  

  金沙js333娱乐场 7

四:split

1.配置

  <property>
    <name>hbase.hregion.max.filesize</name>
    <value>10737418240</value>
  </property>

 

1:配置hosts和ssh无密码登陆

我们在此基础上继续安装hbase 0.92.0

Hadoop+HBase搭建云存储总结 PDF http://www.linuxidc.com/Linux/2013-05/83844.htm

[root@ www.linuxidc.com ~]# cd /opt/hadoop/
一 先下载hbase,因为bug等问题,具体版本请自己选择,这里以 0.92.0为例:
[root@ www.linuxidc.com hadoop]# wget
解压
[root@ www.linuxidc.com hadoop]# tar zxvf hbase-0.92.0.tar.gz 
[root@ www.linuxidc.com hadoop]# mv hbase-0.92.0 hbase_0_92_0
[root@ www.linuxidc.com hadoop]# ln -s hbase_0_92_0 hbase

HBase 结点之间时间不一致造成regionserver启动失败 http://www.linuxidc.com/Linux/2013-06/86655.htm

二 修改配置文件
[root@ www.linuxidc.com hadoop]# vi /etc/profile
在环境变量中添加HBASE_HOME
export HBASE_HOME=/opt/hadoop_1_金沙js333娱乐场 ,0_0/hbase_0_92_0
export PATH=$PATH:$HBASE_HOME/bin
修改HBASE的配置文件
[root@ www.linuxidc.com hadoop]# cd hbase/conf/
1 hbase-env.sh
[root@ www.linuxidc.com conf]# vi hbase-env.sh 
修改参数
export JAVA_HOME=/soft/jdk1.6.0_30
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
export HBASE_PID_DIR=/opt/hadoop_1_0_0/hbase_0_92_0/pids
export HBASE_MANAGES_ZK=true
2 hbase-site.xml
[root@ www.linuxidc.com conf]# vi hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://da-free-test1:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>da-free-test1,da-free-test2,da-free-test3,da-free-test4</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>60000</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>21818</value>
</property>
</configuration>
3 regionservers
[root@ www.linuxidc.com conf]# vi regionservers
da-free-test2
da-free-test3
da-free-test4

Hadoop+ZooKeeper+HBase集群配置 http://www.linuxidc.com/Linux/2013-06/86347.htm

金沙js333娱乐场 8

Hadoop集群安装&HBase实验环境搭建 http://www.linuxidc.com/Linux/2013-04/83560.htm

基于Hadoop集群的HBase集群的配置 http://www.linuxidc.com/Linux/2013-03/80815.htm‘

Hadoop安装部署笔记之-HBase完全分布模式安装 http://www.linuxidc.com/Linux/2012-12/76947.htm

单机版搭建HBase环境图文教程详解 http://www.linuxidc.com/Linux/2012-10/72959.htm 

2:配置hbase

解压tar zxf hbase-0.94.21.tar.gz

配置hbase-env.sh

[linuxidc@linuxidc02 hbase-0.94.21]$ cd conf
[linuxidc@linuxidc02 conf]$ cat hbase-env.sh
#
#/**
# * Copyright 2007 The Apache Software Foundation
# *
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements.  See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership.  The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License.  You may obtain a copy of the License at
# *
# *   
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */

# Set environment variables here.

# This script sets variables multiple times over the course of starting an hbase process,
# so try to keep things idempotent unless you want to take an even deeper look
# into the startup scripts (bin/hbase, etc.)

# The java implementation to use.  Java 1.6 required.
export JAVA_HOME=$HOME/jdk1.7.0_45

# Extra Java CLASSPATH elements.  Optional.
export HBASE_CLASSPATH=$HOME/hadoop/hadoop-1.2.1/conf

# The maximum amount of heap to use, in MB. Default is 1000.
# export HBASE_HEAPSIZE=1000

# Extra Java runtime options.
# Below are what we set by default.  May only work with SUN JVM.
# For more on why as well as other possible settings,
# see
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"

# Uncomment one of the below three options to enable java garbage collection logging for the server-side processes.

# This enables basic gc logging to the .out file.
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

# This enables basic gc logging to its own file.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH>"

# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M"

# Uncomment one of the below three options to enable java garbage collection logging for the client processes.

# This enables basic gc logging to the .out file.
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

# This enables basic gc logging to its own file.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH>"

# This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+.
# If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR .
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<FILE-PATH> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M"

# Uncomment below if you intend to use the EXPERIMENTAL off heap cache.
# export HBASE_OPTS="$HBASE_OPTS -XX:MaxDirectMemorySize="
# Set hbase.offheapcache.percentage in hbase-site.xml to a nonzero value.

# Uncomment and adjust to enable JMX exporting
# See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access.
# More details at:
#
# export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104"

# File naming hosts on which HRegionServers will run.  $HBASE_HOME/conf/regionservers by default.
# export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers

# File naming hosts on which backup HMaster will run.  $HBASE_HOME/conf/backup-masters by default.
# export HBASE_BACKUP_MASTERS=${HBASE_HOME}/conf/backup-masters

# Extra ssh options.  Empty by default.
# export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR"

# Where log files are stored.  $HBASE_HOME/logs by default.
# export HBASE_LOG_DIR=${HBASE_HOME}/logs

# Enable remote JDWP debugging of major HBase processes. Meant for Core Developers
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"

# A string representing this instance of hbase. $USER by default.
# export HBASE_IDENT_STRING=$USER

# The scheduling priority for daemon processes.  See 'man nice'.
# export HBASE_NICENESS=10

# The directory where pid files are stored. /tmp by default.
export HBASE_PID_DIR=/home/linuxidc/pids

# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HBASE_SLAVE_SLEEP=0.1

# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=true

修改的地方
#JDK的安装目录
export JAVA_HOME=$HOME/jdk1.7.0_45
#hadoop的配置目录
export HBASE_CLASSPATH=$HOME/hadoop/hadoop-1.2.1/conf

和最后一行

export HBASE_MANAGES_ZK=true 

#true:表示zookeeper交给hbase管理,启动hbase时,会自动启动hbase-site.xml里的hbase.zookeeper.quorum属性中的所有zookeeper实例 #false:表示启动hbase时,要手动启动所有zookeeper实例

 

配置hbase-site.xml

[linuxidc@linuxidc02 conf]$ cat hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
 * Copyright 2010 The Apache Software Foundation
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *   
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://linuxidc01:9000/user/linuxidc/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
  <name>hbase.master</name>
  <value>linuxidc01:60000</value>
 </property>
 <property>
 <name>hbase.zookeeper.quorum</name>
  <value>linuxidc01,linuxidc02</value>
 </property>

</configuration>

配置regionservers

[linuxidc@linuxidc02 conf]$ cat regionservers
linuxidc01
linuxidc02

更多详情见请继续阅读下一页的精彩内容: http://www.linuxidc.com/Linux/2014-09/106534p2.htm

金沙js333娱乐场 9

本文由js333发布于计算机互联网,转载请注明出处:安装记录,分布式安装

关键词:

上一篇:5新增加硬盘挂载并实现开机自动挂载,4上多路径

下一篇:没有了