通过 docker-compose 快速部署 StarRocks 保姆级教程

一、概述

StarRocks是一个开源的分布式OLAP（在线剖析处理）数据库，旨在供给高功能、低推迟的数据剖析和查询才能。它最初由我国的猎豹移动公司（Cheetah Mobile）开发，并于2016年开源。

StarRocks主要特点和功用包含：

• 列式存储：StarRocks运用列式存储，将数据按列存储在磁盘上，以进步数据的压缩率和查询功能。列式存储有助于削减I/O开支和进步数据扫描效率。

• 分布式架构：StarRocks基于分布式架构设计，能够水平扩展以处理大规模数据集。它支撑数据分片和分布式核算，能够将数据和核算使命分布在多个节点上并行处理。

• 多维数据模型：StarRocks支撑多维数据模型，适用于OLAP剖析和查询。它供给了维度模型、星型模型和雪花模型等，支撑多维查询、聚合和切片切块操作。

• 实时数据同步：StarRocks支撑实时数据同步，能够经过接口、Kafka等方法实时加载数据，并坚持数据的一致性和准确性。

• 高功能查询引擎：StarRocks内置了一个高功能的查询引擎，支撑杂乱的SQL查询操作。它具有快速的查询速度和低推迟，并供给了聚合、排序、分组和衔接等常见的查询操作。

• 高可用性和容错性：StarRocks具有高可用性和容错性，支撑数据的备份和毛病康复。它能够主动处理节点毛病和数据冗余，保证数据的可靠性和可用性。

总的来说，StarRocks是一个面向OLAP场景的高功能分布式数据库，经过列式存储、多维数据模型和高功能查询引擎等特性，供给快速、可扩展的数据剖析和查询才能。它广泛应用于大数据剖析、实时报表、数据仪表盘等场景。

想了解更多关于 StarRocks 可阅读我以下几篇文章：

• 大数据Hadoop之——DorisDB介绍与环境布置（StarRocks）
• 大数据Hadoop之——DorisDB中心概念介绍与简单运用（StarRocks）

从 3.0 版别开端，StarRocks 支撑新的同享数据架构，能够供给更好的可扩展性和更低的成本。

二、前期预备

1）布置 docker

# 装置yum-config-manager装备东西
yum -y install yum-utils
# 主张运用阿里云yum源：（推荐）
#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 装置docker-ce版别
yum install -y docker-ce
# 发动并开机发动
systemctl enable --now docker
docker --version

2）布置 docker-compose

curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version

三、创立网络

# 创立，留意不能运用hadoop_network，要不然发动hs2服务的时分会有问题！！！
docker network create hadoop-network
# 检查
docker network ls

四、StarRocks 编排布置

1）下载 StarRocks 布置包

wget https://releases.starrocks.io/starrocks/StarRocks-3.0.1.tar.gz

2）装备

• conf/fe.conf

LOG_DIR = /opt/apache/StarRocks/fe/log
DATE = "$(date +%Y%m%d-%H%M%S)"
# 修正元数据目录。
meta_dir = /opt/apache/StarRocks/fe/meta
# 在数据库运维中，假如用 IP 作为节点的仅有标识，会出现由于 IP 变动而导致服务不可用的问题。在 2.4 版别中，支撑了 Fully qualified domain name（FQDN）。您能够用域名或结合主机名与端口的方法作为 FE 或 BE 节点的仅有标识，有效避免因 IP 改变导致无法访问的问题。由于容器ip是变动的，所以这里不启用priority_networks。
# priority_networks = 192.168.0.0/24
# 增加 Java 目录
JAVA_HOME = /opt/apache/jdk1.8.0_212
# 修正JVM内存，默认是8G，根据自己机器自定义，默认是-Xmx8192m，这里我修正成Xmx512m，这里有两段装备，jdk 9+运用JAVA_OPTS_FOR_JDK_9
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx512m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$STARROCKS_HOME/log/fe.gc.log.$DATE"
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Dlog4j2.formatMsgNoLookups=true -Xmx512m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$STARROCKS_HOME/log/fe.gc.log.$DATE:time"
sys_log_level = INFO
http_port = 8030
rpc_port = 9020
query_port = 9030
edit_log_port = 9010
# 是否开启 MySQL 服务器的异步 I/O 选项。
mysql_service_nio_enabled = true

• conf/be.conf

sys_log_level = INFO
sys_log_dir = /opt/apache/StarRocks/be/log
be_port = 9060
be_http_port = 8040
heartbeat_service_port = 9050
brpc_port = 8060
# 存储途径
storage_root_path = /opt/apache/StarRocks/be/storage

• conf/apache_hdfs_broker.conf

经过 Broker，StarRocks 可读取对应数据源（如HDFS、S3）上的数据，运用本身的核算资源对数据进行预处理和导入。除此之外，Broker 也被应用于数据导出，备份康复等功用。

broker_ipc_port=8000
client_expire_seconds=300

3）发动脚本 bootstrap.sh

version: '3'
services:
  starrocks-fe-1:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    container_name: starrocks-fe-1
    hostname: starrocks-fe-1
    restart: always
    privileged: true
    env_file:
      - .env
    volumes:
      - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf
    ports:
      - "${StarRocks_FE_HTTP_PORT}"
    expose:
      - "${StarRocks_FE_RPC_PORT}"
      - "${StarRocks_FE_QUERY_PORT}"
      - "${StarRocks_FE_EDIT_LOG_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh fe"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"]
      interval: 10s
      timeout: 20s
      retries: 3
  starrocks-fe-2:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    container_name: starrocks-fe-2
    hostname: starrocks-fe-2
    restart: always
    privileged: true
    env_file:
      - .env
    volumes:
      - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf
    ports:
      - "${StarRocks_FE_HTTP_PORT}"
    expose:
      - "${StarRocks_FE_RPC_PORT}"
      - "${StarRocks_FE_QUERY_PORT}"
      - "${StarRocks_FE_EDIT_LOG_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh fe starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"]
      interval: 10s
      timeout: 20s
      retries: 3
  starrocks-fe-3:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    container_name: starrocks-fe-3
    hostname: starrocks-fe-3
    restart: always
    privileged: true
    env_file:
      - .env
    volumes:
      - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf
    ports:
      - "${StarRocks_FE_HTTP_PORT}"
    expose:
      - "${StarRocks_FE_RPC_PORT}"
      - "${StarRocks_FE_QUERY_PORT}"
      - "${StarRocks_FE_EDIT_LOG_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh fe starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"]
      interval: 10s
      timeout: 20s
      retries: 3
  starrocks-be:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    restart: always
    privileged: true
    deploy:
      replicas: 3
    env_file:
      - .env
    volumes:
      - ./conf/be.conf:${StarRocks_HOME}/be/conf/be.conf
    ports:
      - "${StarRocks_BE_HTTP_PORT}"
    expose:
      - "${StarRocks_BE_PORT}"
      - "${StarRocks_BE_HEARTBEAT_SERVICE_PORT}"
      - "${StarRocks_BE_BRPC_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh be starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_BE_HTTP_PORT} || exit 1"]
      interval: 10s
      timeout: 10s
      retries: 3
  starrocks-apache_hdfs_broker:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    restart: always
    privileged: true
    deploy:
      replicas: 3
    env_file:
      - .env
    volumes:
      - ./conf/apache_hdfs_broker.conf:${StarRocks_HOME}/apache_hdfs_broker/conf/apache_hdfs_broker.conf
    expose:
      - "${StarRocks_BROKER_IPC_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh broker starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_BROKER_IPC_PORT} || exit 1"]
      interval: 10s
      timeout: 10s
      retries: 3
# 衔接外部网络
networks:
  hadoop-network:
    external: true

4）构建镜像 Dockerfile

FROM registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/centos-jdk:7.7.1908
# install client mysql
RUN yum -y install mysql
# 增加 StarRocks 包
ENV StarRocks_VERSION 3.0.1
RUN mkdir /opt/apache/StarRocks-${StarRocks_VERSION}
ADD StarRocks-${StarRocks_VERSION}.tar.gz /opt/apache/
ENV StarRocks_HOME /opt/apache/StarRocks
RUN ln -s /opt/apache/StarRocks-${StarRocks_VERSION} $StarRocks_HOME
# 创立存储目录
RUN mkdir /opt/apache/StarRocks/fe/meta
RUN mkdir /opt/apache/StarRocks/be/storage
# copy bootstrap.sh
COPY bootstrap.sh /opt/apache/
RUN chmod +x /opt/apache/bootstrap.sh
RUN chown -R hadoop:hadoop /opt/apache
WORKDIR $StarRocks_HOME

开端构建镜像

docker build -t registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 . --no-cache
# 为了方便小伙伴下载即可运用，我这里将镜像文件推送到阿里云的镜像库房
docker push registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
### 参数解释
# -t：指定镜像名称
# . ：当时目录Dockerfile
# -f：指定Dockerfile途径
#  --no-cache：不缓存

5）编排 docker-compose.yaml

version: '3'
services:
  starrocks-fe-1:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    container_name: starrocks-fe-1
    hostname: starrocks-fe-1
    restart: always
    privileged: true
    env_file:
      - .env
    volumes:
      - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf
    ports:
      - "${StarRocks_FE_HTTP_PORT}"
    expose:
      - "${StarRocks_FE_RPC_PORT}"
      - "${StarRocks_FE_QUERY_PORT}"
      - "${StarRocks_FE_EDIT_LOG_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh fe"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"]
      interval: 10s
      timeout: 20s
      retries: 3
  starrocks-fe-2:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    container_name: starrocks-fe-2
    hostname: starrocks-fe-2
    restart: always
    privileged: true
    env_file:
      - .env
    volumes:
      - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf
    ports:
      - "${StarRocks_FE_HTTP_PORT}"
    expose:
      - "${StarRocks_FE_RPC_PORT}"
      - "${StarRocks_FE_QUERY_PORT}"
      - "${StarRocks_FE_EDIT_LOG_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh fe starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"]
      interval: 10s
      timeout: 20s
      retries: 3
  starrocks-fe-3:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    container_name: starrocks-fe-3
    hostname: starrocks-fe-3
    restart: always
    privileged: true
    env_file:
      - .env
    volumes:
      - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf
    ports:
      - "${StarRocks_FE_HTTP_PORT}"
    expose:
      - "${StarRocks_FE_RPC_PORT}"
      - "${StarRocks_FE_QUERY_PORT}"
      - "${StarRocks_FE_EDIT_LOG_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh fe starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"]
      interval: 10s
      timeout: 20s
      retries: 3
  starrocks-be:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    restart: always
    privileged: true
    deploy:
      replicas: 3
    env_file:
      - .env
    volumes:
      - ./conf/be.conf:${StarRocks_HOME}/be/conf/be.conf
    ports:
      - "${StarRocks_BE_HTTP_PORT}"
    expose:
      - "${StarRocks_BE_PORT}"
      - "${StarRocks_BE_HEARTBEAT_SERVICE_PORT}"
      - "${StarRocks_BE_BRPC_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh be starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_BE_HTTP_PORT} || exit 1"]
      interval: 10s
      timeout: 10s
      retries: 3
  starrocks-apache_hdfs_broker:
    image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1
    user: "hadoop:hadoop"
    restart: always
    privileged: true
    deploy:
      replicas: 3
    env_file:
      - .env
    volumes:
      - ./conf/apache_hdfs_broker.conf:${StarRocks_HOME}/apache_hdfs_broker/conf/apache_hdfs_broker.conf
    expose:
      - "${StarRocks_BROKER_IPC_PORT}"
    command: ["sh","-c","/opt/apache/bootstrap.sh broker starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"]
    networks:
      - hadoop-network
    healthcheck:
      test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_BROKER_IPC_PORT} || exit 1"]
      interval: 10s
      timeout: 10s
      retries: 3
# 衔接外部网络
networks:
  hadoop-network:
    external: true

.env 文件内容：

StarRocks_HOME=/opt/apache/StarRocks
StarRocks_FE_HTTP_PORT=8030
StarRocks_FE_RPC_PORT=9020
StarRocks_FE_QUERY_PORT=9030
StarRocks_FE_EDIT_LOG_PORT=9010
StarRocks_BE_HTTP_PORT=8040
StarRocks_BE_PORT=9060
StarRocks_BE_HEARTBEAT_SERVICE_PORT=9050
StarRocks_BE_BRPC_PORT=8060
StarRocks_BROKER_IPC_PORT=8000

6）开端布置

# p=sr：项目名，默认项目名是当时目录名称
docker-compose -f docker-compose.yaml -p=sr up -d
# 检查
docker-compose -f docker-compose.yaml -p=sr ps
# 卸载
docker-compose -f docker-compose.yaml -p=sr down

五、简单测验验证

# 检查FE http 对外端口
docker-compose -f docker-compose.yaml -p=sr ps

web：http://ip:port

六、常用的 StarRocks 客户端命令

1）服务启停

### 1、第一个 FE 服务启停
${StarRocks_HOME}/fe/bin/start_fe.sh --host_type FQDN
### 2、增加FE节点
# 增加节点
mysql -h ${fe_leader} -P9030 -uroot -e "ALTER SYSTEM ADD FOLLOWER \"${fqdn}:9010\"";
# 服务发动
${StarRocks_HOME}/fe/bin/start_fe.sh --helper ${fe_leader}:9010 --host_type FQDN
### 3、增加BE节点
# 增加节点
mysql -h ${fe_leader} -P9030 -uroot -e "ALTER SYSTEM ADD BACKEND \"${fqdn}:9050\"";
# 服务发动
${StarRocks_HOME}/be/bin/start_be.sh
#### 4、增加 broker 节点
# 增加节点
mysql -h ${fe_leader} -P9030 -uroot -e "ALTER SYSTEM ADD BROKER ${broker_name} \"${fqdn}:8000\"";
# 服务发动
${StarRocks_HOME}/apache_hdfs_broker/bin/start_broker.sh
### 5、服务停止
${StarRocks_HOME}/fe/bin/stop_fe.sh
${StarRocks_HOME}/be/bin/stop_be.sh
${StarRocks_HOME}/apache_hdfs_broker/bin/stop_broker.sh

2、检查节点状态

mysql -h ${FE} -P9030 -uroot
# 检查 FE
SHOW PROC '/frontends'\G
# 检查 BE
SHOW PROC "/backends"\G
# 检查 broker
SHOW PROC "/brokers"\G

到此 经过 docker-compose 快速布置 StarRocks 保姆级教程就完毕了，后续会持续更新相关技能类文章，有任何疑问欢迎重视我公众号 大数据与云原生技能分享 加群交流或私信沟通，如本篇文章对您有所协助，麻烦帮助一键三连（点赞、转发、收藏）~