1500字范文,内容丰富有趣,写作好帮手!
1500字范文 > 大数据平台架构实战(二)IntelliJ IDEA搭建hadoop

大数据平台架构实战(二)IntelliJ IDEA搭建hadoop

时间:2022-11-13 14:18:07

相关推荐

大数据平台架构实战(二)IntelliJ IDEA搭建hadoop

IntelliJ IDEA 构建工程

工具类开发,随便找,IntelliJ IDEA搭建Hadoop开发环境(下)_Captain.Y.的博客-CSDN博客

生成jar包

由于我们的开发环境和hadoop运行环境都在同一台机器,直接用maven工具就可以了

如果是其他情形 ,尽量保证jar包的依赖完整性。

pom文件

<dependencies><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>2.10.2</version></dependency><!--单元测试依赖--><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.12</version></dependency><dependency><groupId>commons-httpclient</groupId><artifactId>commons-httpclient</artifactId><version>3.1</version></dependency></dependencies><properties><piler.source>1.8</piler.source><piler.target>1.8</piler.target></properties>

打包完成后,就可以在hadoop平台直接运行了

bin/hadoop jar /Users/xxxxxx/Project/hadoop_base/target/hadoop_base-1.0-SNAPSHOT.jar org.xxxx.algorithm.tools.WordCount /input /output/wc

遇到问题

/bin/bash: /bin/java: No such file or directory

22/06/01 15:26:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable22/06/01 15:26:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:803222/06/01 15:26:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.22/06/01 15:26:33 INFO input.FileInputFormat: Total input files to process : 2922/06/01 15:26:34 INFO mapreduce.JobSubmitter: number of splits:2922/06/01 15:26:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1654067025193_000222/06/01 15:26:34 INFO conf.Configuration: resource-types.xml not found22/06/01 15:26:34 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.22/06/01 15:26:34 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE22/06/01 15:26:34 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE22/06/01 15:26:34 INFO impl.YarnClientImpl: Submitted application application_1654067025193_000222/06/01 15:26:34 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1654067025193_0002/22/06/01 15:26:34 INFO mapreduce.Job: Running job: job_1654067025193_000222/06/01 15:26:36 INFO mapreduce.Job: Job job_1654067025193_0002 running in uber mode : false22/06/01 15:26:36 INFO mapreduce.Job: map 0% reduce 0%22/06/01 15:26:36 INFO mapreduce.Job: Job job_1654067025193_0002 failed with state FAILED due to: Application application_1654067025193_0002 failed 2 times due to AM Container for appattempt_1654067025193_0002_000002 exited with exitCode: 127Failing this attempt.Diagnostics: [-06-01 15:26:36.146]Exception from container-launch.Container id: container_1654067025193_0002_02_000001Exit code: 127[-06-01 15:26:36.147]Container exited with a non-zero exit code 127. Error file: prelaunch.err.Last 4096 bytes of prelaunch.err :Last 4096 bytes of stderr :/bin/bash: /bin/java: No such file or directory[-06-01 15:26:36.147]Container exited with a non-zero exit code 127. Error file: prelaunch.err.Last 4096 bytes of prelaunch.err :Last 4096 bytes of stderr :/bin/bash: /bin/java: No such file or directoryFor more detailed output, check the application tracking page: http://localhost:8088/cluster/app/application_1654067025193_0002 Then click on links to logs of each attempt.. Failing the application.22/06/01 15:26:36 INFO mapreduce.Job: Counters: 0

解决方法:

修改etc/hadoop/hadoop-env.sh

export JAVA_HOME=/path/to/java/home

本内容不代表本网观点和政治立场,如有侵犯你的权益请联系我们处理。
网友评论
网友评论仅供其表达个人看法,并不表明网站立场。