ONEFENG

flink集群的搭建

2021/12/12 flink 共 1450 字，约 5 分钟

flink集群的搭建

设置免密登录

设置主节点到从节点免密登录。首先在主节点执行

ssh-keygen -t rsa

在从节点分别执行，根据提示输入对应密码。

cat ~/.ssh/id_rsa.pub | ssh release_env@10.0.0.133 "cat - >> ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh release_env@10.10.1.24 "cat - >> ~/.ssh/authorized_keys"

flink部署

在主节点的/home/release_env目录下载flink

wget https://archive.apache.org/dist/flink/flink-1.12.0/flink-1.12.0-bin-scala_2.12.tgz

解压文件

tar -zxvf flink-1.12.0-bin-scala_2.12.tgz

修改conf/flink-conf.yaml，jobmanager.rpc.address地址为主节点ip，numberOfTaskSlots改为12

jobmanager.rpc.address: 10.0.0.139
taskmanager.numberOfTaskSlots: 12

修改conf/master为主节点地址

10.0.0.139:8081

修改conf/workers为任务节点地址

10.0.0.133
10.10.1.24

将解压修改后的文件复制到两个从节点

scp -r /home/release_env/flink-1.12.0 release_env@10.0.0.133:/home/release_env
scp -r /home/release_env/flink-1.12.0 release_env@10.10.1.24:/home/release_env

在主节点的/home/release_env/flink-1.12.0/bin目录下，启动flink集群

./start-cluster.sh

docker部署

创建排版docker-stack.yml

version: "3"
services:
  jobmanager:
    image: flink:1.12.0-scala_2.12
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
    volumes:
      - flink_job_config:/opt/flink/conf


  taskmanager1:
    image: flink:1.12.0-scala_2.12
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

    volumes:
      - flink_task_config:/opt/flink/conf

    deploy:
      replicas: 5

volumes:
  flink_job_config:
  flink_task_config:

部署stack

docker  stack deploy  -c ./docker-stack.yml --prune --resolve-image always  flink-service

搜索

    内容