一、概述
StarRocks是一个开源的分布式OLAP(在线剖析处理)数据库,旨在供给高功能、低推迟的数据剖析和查询才能。它最初由我国的猎豹移动公司(Cheetah Mobile)开发,并于2016年开源。
StarRocks主要特点和功用包含:
-
列式存储:StarRocks运用列式存储,将数据按列存储在磁盘上,以进步数据的压缩率和查询功能。列式存储有助于削减I/O开支和进步数据扫描效率。
-
分布式架构:StarRocks基于分布式架构设计,能够水平扩展以处理大规模数据集。它支撑数据分片和分布式核算,能够将数据和核算使命分布在多个节点上并行处理。
-
多维数据模型:StarRocks支撑多维数据模型,适用于OLAP剖析和查询。它供给了维度模型、星型模型和雪花模型等,支撑多维查询、聚合和切片切块操作。
-
实时数据同步:StarRocks支撑实时数据同步,能够经过接口、Kafka等方法实时加载数据,并坚持数据的一致性和准确性。
-
高功能查询引擎:StarRocks内置了一个高功能的查询引擎,支撑杂乱的SQL查询操作。它具有快速的查询速度和低推迟,并供给了聚合、排序、分组和衔接等常见的查询操作。
-
高可用性和容错性:StarRocks具有高可用性和容错性,支撑数据的备份和毛病康复。它能够主动处理节点毛病和数据冗余,保证数据的可靠性和可用性。
总的来说,StarRocks是一个面向OLAP场景的高功能分布式数据库,经过列式存储、多维数据模型和高功能查询引擎等特性,供给快速、可扩展的数据剖析和查询才能。它广泛应用于大数据剖析、实时报表、数据仪表盘等场景。

- 大数据Hadoop之——DorisDB介绍与环境布置(StarRocks)
- 大数据Hadoop之——DorisDB中心概念介绍与简单运用(StarRocks)
从 3.0 版别开端,StarRocks 支撑新的同享数据架构,能够供给更好的可扩展性和更低的成本。

二、前期预备
1)布置 docker
# 装置yum-config-manager装备东西 yum -y install yum-utils # 主张运用阿里云yum源:(推荐) #yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 装置docker-ce版别 yum install -y docker-ce # 发动并开机发动 systemctl enable --now docker docker --version
2)布置 docker-compose
curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose docker-compose --version
三、创立网络
# 创立,留意不能运用hadoop_network,要不然发动hs2服务的时分会有问题!!! docker network create hadoop-network # 检查 docker network ls
四、StarRocks 编排布置
1)下载 StarRocks 布置包
wget https://releases.starrocks.io/starrocks/StarRocks-3.0.1.tar.gz
2)装备
conf/fe.conf
LOG_DIR = /opt/apache/StarRocks/fe/log DATE = "$(date +%Y%m%d-%H%M%S)" # 修正元数据目录。 meta_dir = /opt/apache/StarRocks/fe/meta # 在数据库运维中,假如用 IP 作为节点的仅有标识,会出现由于 IP 变动而导致服务不可用的问题。在 2.4 版别中,支撑了 Fully qualified domain name(FQDN)。您能够用域名或结合主机名与端口的方法作为 FE 或 BE 节点的仅有标识,有效避免因 IP 改变导致无法访问的问题。由于容器ip是变动的,所以这里不启用priority_networks。 # priority_networks = 192.168.0.0/24 # 增加 Java 目录 JAVA_HOME = /opt/apache/jdk1.8.0_212 # 修正JVM内存,默认是8G,根据自己机器自定义,默认是-Xmx8192m,这里我修正成Xmx512m,这里有两段装备,jdk 9+运用JAVA_OPTS_FOR_JDK_9 JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx512m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$STARROCKS_HOME/log/fe.gc.log.$DATE" # For jdk 9+, this JAVA_OPTS will be used as default JVM options JAVA_OPTS_FOR_JDK_9="-Dlog4j2.formatMsgNoLookups=true -Xmx512m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$STARROCKS_HOME/log/fe.gc.log.$DATE:time" sys_log_level = INFO http_port = 8030 rpc_port = 9020 query_port = 9030 edit_log_port = 9010 # 是否开启 MySQL 服务器的异步 I/O 选项。 mysql_service_nio_enabled = true
conf/be.conf
sys_log_level = INFO sys_log_dir = /opt/apache/StarRocks/be/log be_port = 9060 be_http_port = 8040 heartbeat_service_port = 9050 brpc_port = 8060 # 存储途径 storage_root_path = /opt/apache/StarRocks/be/storage
conf/apache_hdfs_broker.conf
经过 Broker,StarRocks 可读取对应数据源(如HDFS、S3)上的数据,运用本身的核算资源对数据进行预处理和导入。除此之外,Broker 也被应用于数据导出,备份康复等功用。
broker_ipc_port=8000 client_expire_seconds=300
3)发动脚本 bootstrap.sh
version: '3' services: starrocks-fe-1: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" container_name: starrocks-fe-1 hostname: starrocks-fe-1 restart: always privileged: true env_file: - .env volumes: - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf ports: - "${StarRocks_FE_HTTP_PORT}" expose: - "${StarRocks_FE_RPC_PORT}" - "${StarRocks_FE_QUERY_PORT}" - "${StarRocks_FE_EDIT_LOG_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh fe"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"] interval: 10s timeout: 20s retries: 3 starrocks-fe-2: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" container_name: starrocks-fe-2 hostname: starrocks-fe-2 restart: always privileged: true env_file: - .env volumes: - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf ports: - "${StarRocks_FE_HTTP_PORT}" expose: - "${StarRocks_FE_RPC_PORT}" - "${StarRocks_FE_QUERY_PORT}" - "${StarRocks_FE_EDIT_LOG_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh fe starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"] interval: 10s timeout: 20s retries: 3 starrocks-fe-3: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" container_name: starrocks-fe-3 hostname: starrocks-fe-3 restart: always privileged: true env_file: - .env volumes: - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf ports: - "${StarRocks_FE_HTTP_PORT}" expose: - "${StarRocks_FE_RPC_PORT}" - "${StarRocks_FE_QUERY_PORT}" - "${StarRocks_FE_EDIT_LOG_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh fe starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"] interval: 10s timeout: 20s retries: 3 starrocks-be: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" restart: always privileged: true deploy: replicas: 3 env_file: - .env volumes: - ./conf/be.conf:${StarRocks_HOME}/be/conf/be.conf ports: - "${StarRocks_BE_HTTP_PORT}" expose: - "${StarRocks_BE_PORT}" - "${StarRocks_BE_HEARTBEAT_SERVICE_PORT}" - "${StarRocks_BE_BRPC_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh be starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_BE_HTTP_PORT} || exit 1"] interval: 10s timeout: 10s retries: 3 starrocks-apache_hdfs_broker: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" restart: always privileged: true deploy: replicas: 3 env_file: - .env volumes: - ./conf/apache_hdfs_broker.conf:${StarRocks_HOME}/apache_hdfs_broker/conf/apache_hdfs_broker.conf expose: - "${StarRocks_BROKER_IPC_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh broker starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_BROKER_IPC_PORT} || exit 1"] interval: 10s timeout: 10s retries: 3 # 衔接外部网络 networks: hadoop-network: external: true
4)构建镜像 Dockerfile
FROM registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/centos-jdk:7.7.1908 # install client mysql RUN yum -y install mysql # 增加 StarRocks 包 ENV StarRocks_VERSION 3.0.1 RUN mkdir /opt/apache/StarRocks-${StarRocks_VERSION} ADD StarRocks-${StarRocks_VERSION}.tar.gz /opt/apache/ ENV StarRocks_HOME /opt/apache/StarRocks RUN ln -s /opt/apache/StarRocks-${StarRocks_VERSION} $StarRocks_HOME # 创立存储目录 RUN mkdir /opt/apache/StarRocks/fe/meta RUN mkdir /opt/apache/StarRocks/be/storage # copy bootstrap.sh COPY bootstrap.sh /opt/apache/ RUN chmod +x /opt/apache/bootstrap.sh RUN chown -R hadoop:hadoop /opt/apache WORKDIR $StarRocks_HOME
开端构建镜像
docker build -t registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 . --no-cache # 为了方便小伙伴下载即可运用,我这里将镜像文件推送到阿里云的镜像库房 docker push registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 ### 参数解释 # -t:指定镜像名称 # . :当时目录Dockerfile # -f:指定Dockerfile途径 # --no-cache:不缓存
5)编排 docker-compose.yaml
version: '3' services: starrocks-fe-1: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" container_name: starrocks-fe-1 hostname: starrocks-fe-1 restart: always privileged: true env_file: - .env volumes: - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf ports: - "${StarRocks_FE_HTTP_PORT}" expose: - "${StarRocks_FE_RPC_PORT}" - "${StarRocks_FE_QUERY_PORT}" - "${StarRocks_FE_EDIT_LOG_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh fe"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"] interval: 10s timeout: 20s retries: 3 starrocks-fe-2: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" container_name: starrocks-fe-2 hostname: starrocks-fe-2 restart: always privileged: true env_file: - .env volumes: - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf ports: - "${StarRocks_FE_HTTP_PORT}" expose: - "${StarRocks_FE_RPC_PORT}" - "${StarRocks_FE_QUERY_PORT}" - "${StarRocks_FE_EDIT_LOG_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh fe starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"] interval: 10s timeout: 20s retries: 3 starrocks-fe-3: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" container_name: starrocks-fe-3 hostname: starrocks-fe-3 restart: always privileged: true env_file: - .env volumes: - ./conf/fe.conf:${StarRocks_HOME}/fe/conf/fe.conf ports: - "${StarRocks_FE_HTTP_PORT}" expose: - "${StarRocks_FE_RPC_PORT}" - "${StarRocks_FE_QUERY_PORT}" - "${StarRocks_FE_EDIT_LOG_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh fe starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_FE_HTTP_PORT} || exit 1"] interval: 10s timeout: 20s retries: 3 starrocks-be: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" restart: always privileged: true deploy: replicas: 3 env_file: - .env volumes: - ./conf/be.conf:${StarRocks_HOME}/be/conf/be.conf ports: - "${StarRocks_BE_HTTP_PORT}" expose: - "${StarRocks_BE_PORT}" - "${StarRocks_BE_HEARTBEAT_SERVICE_PORT}" - "${StarRocks_BE_BRPC_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh be starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_BE_HTTP_PORT} || exit 1"] interval: 10s timeout: 10s retries: 3 starrocks-apache_hdfs_broker: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/starrocks:3.0.1 user: "hadoop:hadoop" restart: always privileged: true deploy: replicas: 3 env_file: - .env volumes: - ./conf/apache_hdfs_broker.conf:${StarRocks_HOME}/apache_hdfs_broker/conf/apache_hdfs_broker.conf expose: - "${StarRocks_BROKER_IPC_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh broker starrocks-fe-1 ${StarRocks_FE_QUERY_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${StarRocks_BROKER_IPC_PORT} || exit 1"] interval: 10s timeout: 10s retries: 3 # 衔接外部网络 networks: hadoop-network: external: true
.env
文件内容:
StarRocks_HOME=/opt/apache/StarRocks StarRocks_FE_HTTP_PORT=8030 StarRocks_FE_RPC_PORT=9020 StarRocks_FE_QUERY_PORT=9030 StarRocks_FE_EDIT_LOG_PORT=9010 StarRocks_BE_HTTP_PORT=8040 StarRocks_BE_PORT=9060 StarRocks_BE_HEARTBEAT_SERVICE_PORT=9050 StarRocks_BE_BRPC_PORT=8060 StarRocks_BROKER_IPC_PORT=8000
6)开端布置
# p=sr:项目名,默认项目名是当时目录名称 docker-compose -f docker-compose.yaml -p=sr up -d # 检查 docker-compose -f docker-compose.yaml -p=sr ps # 卸载 docker-compose -f docker-compose.yaml -p=sr down

五、简单测验验证
# 检查FE http 对外端口 docker-compose -f docker-compose.yaml -p=sr ps
web:http://ip:port

六、常用的 StarRocks 客户端命令
1)服务启停
### 1、第一个 FE 服务启停 ${StarRocks_HOME}/fe/bin/start_fe.sh --host_type FQDN ### 2、增加FE节点 # 增加节点 mysql -h ${fe_leader} -P9030 -uroot -e "ALTER SYSTEM ADD FOLLOWER \"${fqdn}:9010\""; # 服务发动 ${StarRocks_HOME}/fe/bin/start_fe.sh --helper ${fe_leader}:9010 --host_type FQDN ### 3、增加BE节点 # 增加节点 mysql -h ${fe_leader} -P9030 -uroot -e "ALTER SYSTEM ADD BACKEND \"${fqdn}:9050\""; # 服务发动 ${StarRocks_HOME}/be/bin/start_be.sh #### 4、增加 broker 节点 # 增加节点 mysql -h ${fe_leader} -P9030 -uroot -e "ALTER SYSTEM ADD BROKER ${broker_name} \"${fqdn}:8000\""; # 服务发动 ${StarRocks_HOME}/apache_hdfs_broker/bin/start_broker.sh ### 5、服务停止 ${StarRocks_HOME}/fe/bin/stop_fe.sh ${StarRocks_HOME}/be/bin/stop_be.sh ${StarRocks_HOME}/apache_hdfs_broker/bin/stop_broker.sh
2、检查节点状态
mysql -h ${FE} -P9030 -uroot # 检查 FE SHOW PROC '/frontends'\G # 检查 BE SHOW PROC "/backends"\G # 检查 broker SHOW PROC "/brokers"\G
到此 经过 docker-compose 快速布置 StarRocks 保姆级教程就完毕了,后续会持续更新相关技能类文章,有任何疑问欢迎重视我公众号 大数据与云原生技能分享
加群交流或私信沟通,如本篇文章对您有所协助,麻烦帮助一键三连(点赞、转发、收藏)~
