wordpress幻灯片修改,群站优化之链轮模式,网页代码编辑器,vps如何建两个网站文章目录0. 常规解压安装#xff0c;并添加环境变量1. 下载并覆盖 bin 文件夹2. 使VERSION文件的clusterID一致3. 贴下单机配置4. 测试 Hadoop5. 安装Spark环境#xff1a;win 10 java 1.8.0_281 Scala 2.11.11 Hadoop 2.7.7 Spark2.4.70. 常规解压安装#xff0c;并添加…
文章目录0. 常规解压安装并添加环境变量1. 下载并覆盖 bin 文件夹2. 使VERSION文件的clusterID一致3. 贴下单机配置4. 测试 Hadoop5. 安装Spark环境win 10 java 1.8.0_281 Scala 2.11.11 Hadoop 2.7.7 Spark2.4.70. 常规解压安装并添加环境变量 1. 下载并覆盖 bin 文件夹
下载 hadooponwindows-master.zip 下载地址https://pan.baidu.com/s/1o7YTlJO 将下载好的 hadooponwindows-master.zip 解压将解压后的 bin目录下的所有文件直接覆盖Hadoop的 bin目录
2. 使VERSION文件的clusterID一致
Datanode启动问题 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool
原因是clusterID不一致删除tmp下的内容从/home/hdp/hadoop/name/current/VERSION 获得clusterID修改到/home/hdp/hadoop/data/current/VERSION修改保持一致然后重启服务3. 贴下单机配置
core-site.xml
configurationpropertynamefs.default.name/namevaluehdfs://localhost:9000/value/propertypropertynamehadoop.tmp.dir/namevalue/D:/hadoop-2.7.7/data/tmp/value/property
/configurationhdfs-site.xml
configurationpropertynamedfs.replication/namevalue1/value/propertyproperty namedfs.namenode.name.dir/name valuefile:/D:/hadoop-2.7.7/data/namenode/value/property property namedfs.datanode.data.dir/name valuefile:/D:/hadoop-2.7.7/data/datanode/value/property/configurationmapred-site.xml
configurationpropertynamemapreduce.job.user.name/namevalue%USERNAME%/value/propertypropertynamemapreduce.framework.name/namevalueyarn/value/propertypropertynameyarn.apps.stagingDir/namevalue/user/%USERNAME%/staging/value/propertypropertynamemapreduce.jobtracker.address/namevaluelocal/value/property
/configurationyarn-site.xml
configuration!-- nodemanager要求的内存最低为1024 --propertynameyarn.nodemanager.resource.memory-mb/namevalue1024/value/propertypropertynameyarn.nodemanager.resource.cpu-vcores/namevalue1/value/property
/configurationD:\hadoop-2.7.7\etc\hadoop\hadoop-env.cmd
更改1行
set JAVA_HOMED:\Java\jdk1.8.0_281末尾追加4行
set HADOOP_PREFIX%HADOOP_HOME%
set HADOOP_CONF_DIR%HADOOP_PREFIX%\etc\hadoop
set YARN_CONF_DIR%HADOOP_CONF_DIR%
set PATH%PATH%;%HADOOP_PREFIX%\bin4. 测试 Hadoop
格式化启动
hadoop namenode -format
start-dfs.cmd
start-yarn.cmdjps 查看进程 网页能打开
http://localhost:8088/cluster
http://localhost:50070
创建 hdfs 文件夹并查看
hadoop fs -mkdir hdfs://localhost:9000/user/
hdfs dfs -ls /5. 安装Spark
参考https://blog.csdn.net/weixin_45092662/article/details/107490615
添加 环境变量 Path
PS C:\Users\xxx spark-shell.cmd
Using Sparks default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to WARN.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://windows10.microdone.cn:4040
Spark context available as sc (master local[*], app id local-1619526523582).
Spark session available as spark.
Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ / __/ _//___/ .__/\_,_/_/ /_/\_\ version 2.4.7/_/Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_281)
Type in expressions to have them evaluated.
Type :help for more information.
