一、概述
NGINX 署理对错常常见的,内网生成环境一般不让在本机直接拜访,这时候就得利用署理,可是一般署理就得脱敏或去掉页面上的高危操作。这儿就拿 YARN 署理来解说。这儿也会涉及到 subs_filter
解说,假如对 subs_filter
不理解的小伙伴,能够先查阅我这篇文章:NGINX sub_filter和subs_filter 指令解说
二、布置 Hadoop 环境
Hadoop 的布置能够参阅我之前的以下几篇文章:
- 大数据Hadoop原理介绍 装置 实战操作(HDFS YARN MapReduce)
- 大数据Hadoop之——Hadoop 3.3.4 HA(高可用)原理与实现(QJM)
- Hadoop on k8s 编列布置进阶篇
- Hadoop HA on k8s 编列布置进阶篇
- 经过 docker-compose 快速布置 Hadoop 集群极简教程
三、布置 yarn-proxy on docker
这儿布置的是根据GitHub上的一个项目:github.com/HBigdata/ya…
1)布置 docker
# 装置yum-config-manager装备东西
yum -y install yum-utils
# 建议使用阿里云yum源:(引荐)
#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 装置docker-ce版本
yum install -y docker-ce
# 启动并开机启动
systemctl enable --now docker
docker --version
2)布置 docker-compose
curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
chmod x /usr/local/bin/docker-compose
docker-compose --version
3)布置 dnsmasq
-
dnsmasq
先去解析/etc/hosts
文件。 - 再去解析
/etc/dnsmasq.d/
下的*.conf
文件,而且这些文件的优先级要高于/etc/dnsmasq.conf
。 - 我们自定义的
resolv.dnsmasq.conf
中的DNS
也被称为上游DNS
,这是最后去查询解析的。
yum install -y dnsmasq
systemctl start dnsmasq.service
把 hadoop
相关的主机名映射都提早装备到 /etc/hosts
文件中。
想了解更多,能够参阅我这篇文章:dnsmasq( DNS和DHCP)服务
4)下载 yarn-proxy
git clone https://github.com/HBigdata/yarn-reverse-proxy.git
cd yarn-reverse-proxy-master
cp docker-compose.example.yml docker-compose.yml
5)修正装备
version: '3'
services:
app:
build: .
#expose:
# - 80
## WARNING: uncomment the following only if you are
## not directly exposing a non-kerberized cluster
## to the Internet
# ports:
# - "5555:80"
environment:
TZ : 'Asia/Shanghai'
dns:
- localhost
ports:
- "80:80"
# 指定 DNS 服务
environment:
# Because your primary resource-manager is at /rm1
# access your main status page at $SERVER_URL_BASE/rm1
SERVERS: "hostname-rm-110:8088@rm110
hostname-rm-111:8088@rm111
hostname-nm-112:8042@nm112
hostname-nm-113:8042@nm113
hostname-nm-114:8042@nm114
hostname-nm-115:8042@nm115
hostname-nm-116:8042@nm116
hostname-nm-117:8042@nm117
hostname-nm-118:8042@nm118
hostname-jn-119:19888@jh119"
RESOLVER: YOUR_NETWORKS_DNS_SERVER # or 127.0.0.11 for the system resolver
SERVER_URL_BASE: http://REVERSE_PROXY_HOSTNAME:80
需求修正的地方:
-
SERVERS
、REVERSE_PROXY_HOSTNAME
-
dns
地址记住更好成自己的dns
机器。
6)开始布置
docker-compose build
docker-compose up -d
7)经过 subs_filter 进行改善
假如nm节点太多了,都遍历署理的话,就导致超过NGINX装备的长度的,太长装备也影响阅读,所以这儿就经过 nginx subs_filter
进行改善。
subs_filter
的布置与用法能够参阅我这篇文章:NGINX sub_filter和subs_filter 指令解说
经过正则表达式修正 run.sh
# 将这段内容替换成正则
cat <<EOF
# Rewrite references without the scheme, e.g. //google.com
sub_filter 'src="https://$REWRITE_NAME:$REWRITE_PORT' 'src="https://juejin.im/post/7328670028604801036/$SERVER_URL_BASE/$REWRITE_ALIAS';
sub_filter 'href="https://$REWRITE_NAME:$REWRITE_PORT' 'href="https://juejin.im/post/7328670028604801036/$SERVER_URL_BASE/$REWRITE_ALIAS';
sub_filter 'href='//$REWRITE_NAME:$REWRITE_PORT' 'href='$SERVER_URL_BASE/$REWRITE_ALIAS';
# Rewrite all absolute references, regardless of where
# they appear in the page
sub_filter 'http://$REWRITE_NAME:$REWRITE_PORT/' '$SERVER_URL_BASE/$REWRITE_ALIAS/';
sub_filter 'http://$REWRITE_NAME:$REWRITE_PORT' '$SERVER_URL_BASE/$REWRITE_ALIAS';
proxy_redirect http://$REWRITE_NAME:$REWRITE_PORT/ $SERVER_URL_BASE/$REWRITE_ALIAS/;
EOF
# 示例替换
cat <<EOF
# Rewrite references without the scheme, e.g. //google.com
sub_filter 'src="https://local-168-(w )-(d ):$REWRITE_PORT' 'src="https://juejin.im/post/7328670028604801036/$SERVER_URL_BASE/$1$2' g r;
sub_filter 'href="https://local-168-(w )-(d ):$REWRITE_PORT' 'href="https://juejin.im/post/7328670028604801036/$SERVER_URL_BASE/$1$2' g r;
sub_filter 'href='//local-168-(w )-(d ):$REWRITE_PORT' 'href='$SERVER_URL_BASE/$1$2' g r;
# Rewrite all absolute references, regardless of where
# they appear in the page
sub_filter 'http://local-168-(w )-(d ):$REWRITE_PORT/' '$SERVER_URL_BASE/$1$2/' g r;
sub_filter 'http://local-168-(w )-(d ):$REWRITE_PORT' '$SERVER_URL_BASE/$1$2' g r;
proxy_redirect http://local-168-(w )-(d ):$REWRITE_PORT/ $SERVER_URL_BASE/$1$2/ g r;
EOF
四、布置 yarn-proxy on k8s
yarn-proxy on k8s
布置跟 docker-compose
差不多,这儿就不再重复解说了,有任何疑问欢迎给我留言或私信。
NGINX 实战操作(yarn-proxy)解说就先到这儿了,有任何疑问也可关注我大众号:大数据与云原生技术共享,进行技术交流,如本篇文章对您有所协助,麻烦帮忙一键三连(点赞、转发、收藏)~