二十条优化措施,seo的主要内容,信誉好的电商网站建设,上海市政大厅官网1、创建Headless Service服务
Headless 服务类型并不分配容器云虚拟 IP#xff0c;而是直接暴露所属 Pod 的 DNS 记录。没有默认负载均衡器#xff0c;可直接访问 Pod IP 地址。因此#xff0c;当我们需要与集群内真实的 Pod IP 地址进行直接交互时#xff0c;Headless 服…1、创建Headless Service服务
Headless 服务类型并不分配容器云虚拟 IP而是直接暴露所属 Pod 的 DNS 记录。没有默认负载均衡器可直接访问 Pod IP 地址。因此当我们需要与集群内真实的 Pod IP 地址进行直接交互时Headless 服务就很有用。 其中Service的关键配置如下clusterIP: None不让其获取clusterIP DNS解析的时候直接走pod。
---
kind: Service
apiVersion: v1
metadata:name: ecc-spark-servicenamespace: ecc-spark-cluster
spec:clusterIP: Noneports:- port: 7077protocol: TCPtargetPort: 7077name: spark- port: 10000protocol: TCPtargetPort: 10000name: thrift-server-tcp- port: 8080targetPort: 8080name: http- port: 45970protocol: TCPtargetPort: 45970name: thrift-server-driver-tcp - port: 45980protocol: TCPtargetPort: 45980name: thrift-server-blockmanager-tcp - port: 4040protocol: TCPtargetPort: 4040name: thrift-server-tasks-tcp selector:app: ecc-spark-serviceEOFService的完全域名 ecc-spark-service.ecc-spark-cluster.svc.cluster.local headless service的完全域名: headless-service.ecc-spark-cluster.svc.cluster.local 在容器里面ping 完全域名 service解析出的地址是clusterIPheadless service 解析出来的地址是 pod IP。
2、构建spark集群
2.1 、创建spark master
spark master分为两个部分一个是类型为ReplicationController的主体命名为ecc-spark-master.yaml另一部分为一个service暴露master的7077端口给slave使用。
#如下是把thriftserver部署在master节点则需要暴露thriftserver端口、driver端口、
#blockmanager端口服务,以提供worker节点executor与driver交互.
cat ecc-spark-master.yaml EOF
kind: Deployment
apiVersion: apps/v1
metadata:name: ecc-spark-masternamespace: ecc-spark-clusterlabels:app: ecc-spark-master
spec:replicas: 1selector:matchLabels:app: ecc-spark-mastertemplate:metadata:labels:app: ecc-spark-masterspec:serviceAccountName: spark-cdpsecurityContext: {}dnsPolicy: ClusterFirsthostname: ecc-spark-mastercontainers:- name: ecc-spark-masterimage: spark:3.4.1imagePullPolicy: IfNotPresentcommand: [/bin/sh]args: [-c,sh /opt/spark/sbin/start-master.sh tail -f /opt/spark/logs/spark--org.apache.spark.deploy.master.Master-1-*]ports:- containerPort: 7077- containerPort: 8080volumeMounts:- mountPath: /opt/usrjars/name: ecc-spark-pvclivenessProbe:failureThreshold: 9initialDelaySeconds: 2periodSeconds: 15successThreshold: 1tcpSocket:port: 8080timeoutSeconds: 10resources:requests:cpu: 2memory: 6Gilimits:cpu: 2memory: 6Gi- env:- SPARK_LOCAL_DIRSvalue: /odsdata/sparkdirs/ volumes:- name: ecc-spark-pvcpersistentVolumeClaim:claimName: ecc-spark-pvc-static2.2、创建spark worker
在启动spark worker脚本中需要传入master的地址在容器云kubernetes dns且设置了service的缘故可以通过ecc-spark-master.ecc-spark-cluster.svc.cluster.local:7077访问。
cat ecc-spark-worker.yaml EOF
kind: Deployment
apiVersion: apps/v1
metadata:name: ecc-spark-workernamespace: ecc-spark-clusterlabels:app: ecc-spark-worker
spec:replicas: 1selector:matchLabels:app: ecc-spark-workertemplate:metadata:labels:app: ecc-spark-workerspec:serviceAccountName: spark-cdpsecurityContext: {}dnsPolicy: ClusterFirsthostname: ecc-spark-workercontainers:- name: ecc-spark-workerimage: spark:3.4.1imagePullPolicy: IfNotPresentcommand: [/bin/sh]args: [-c,sh /opt/spark/sbin/start-worker.sh spark://ecc-spark-master.ecc-spark-cluster.svc.cluster.local:7077;tail -f /opt/spark/logs/spark--org.apache.spark.deploy.worker.Worker*]ports:- containerPort: 8081volumeMounts:- mountPath: /opt/usrjars/name: ecc-spark-pvcresources:requests:cpu: 2memory: 2Gilimits:cpu: 2memory: 4Gi- env:- SPARK_LOCAL_DIRSvalue: /odsdata/sparkdirs/ volumes:- name: ecc-spark-pvcpersistentVolumeClaim:claimName: ecc-spark-pvc-staticEOF2.3 构建pyspark提交环境
import json
import flask
from flask import Flask
from concurrent.futures import ThreadPoolExecutorapp Flask(__name__)
pool ThreadPoolExecutor(max_workers8)app.route(/)
def hello_world(): # put applications code herereturn Hello World!app.route(/downloadCode, methods[post])
def download_file():model_id flask.request.json.get(modelId)print(model_id)异步提交任务pool.submit()return json.dumps(0, ensure_asciiFalse)app.route(/modelRun, methods[post])
def model_run():异步提交任务pool.submit()return json.dumps(0, ensure_asciiFalse)if __name__ __main__:app.run()sparkc67e6477b2f1:/opt/spark$ python3
Python 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0] on linux
Type help, copyright, credits or license for more information.将python的调用整合到:start-master.sh 文件末尾启动调用便可以通过k8s暴露spark-master的F5端口实现http调用。
3、使用spark-operator安装spark集群方式
可以参考阿里云文章搭建Spark应用
