CentOS6.5下Hadoop的编译示例-创新互联

这篇文章给大家分享的是有关CentOS6.5下Hadoop的编译示例的内容。小编觉得挺实用的，因此分享给大家做个参考，一起跟随小编过来看看吧。

Hadoop介绍
(1) hadoop.apache.org
(2) 狭义: 软件(HDFS、MapReduce、Yarn)
   褒义: 以hadoop为主的生态圈


(3)Hadoop1.x
HDFS  : 分布式文件系统，存储
MapReduce: 分布式计算+资源调度和作业调度


(4)Hadoop2.x
HDFS :  存储
MapReduce : 计算
Yarn : YetAnother Resource Negotiator  资源调度和作业调度平台

1.检查是否有无正在运行的hadoop.

ps –ef|grep hadoop      find / -name  hadoop

2在/opt目录下创建sourcecode  software 两个文件夹。

3.上传hadoop的tar包到sourcecode.

4.解压hadoop-2.8.1-src.tar.gz。

tar  -xzvf  hadoop-2.8.1-src.tar.gz 

5.查看Hadoop编译的环境

cd  hadoop-2.8.1-src>>cat BUILDING.txt 

6.安装JDK

7.配置JDK环境变量在/etc/profile

8上传maven包

9.配置maven环境变量---/etc/profile

10.上传安装protobuf-2.5.0.tar

11Findbugs安装
[root@hadoop01 software]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring findbugs-1.3.9.zip...
  100%    7369 KB    7369 KB/sec    00:00:01       0 Errors  


[root@hadoop01 software]# unzip findbugs-1.3.9.zip


[root@hadoop01 software]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop01 software]# 
[root@hadoop01 software]# source /etc/profile
[root@hadoop01 software]# findbugs -version
1.3.9
[root@hadoop001 software]# 

12.编译
[root@hadoop01 sourcecode]# cd hadoop-2.8.1-src
[root@hadoop01 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  7.770 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  4.990 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  4.920 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 12.936 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.809 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  5.013 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 17.617 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 23.551 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 26.352 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 10.305 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [05:58 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 19.105 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 27.479 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.174 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:32 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [06:42 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 20.370 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 40.720 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 21.298 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 11.818 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.148 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.192 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 41.517 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:19 min]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.192 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 19.421 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 42.398 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  8.925 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 16.120 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 57.415 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  3.869 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 14.325 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 11.814 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 10.027 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.276 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  8.333 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  5.473 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.160 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 13.204 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  8.106 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.514 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [01:09 min]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 40.479 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 10.304 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 27.335 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 19.910 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 16.657 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  4.591 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 12.346 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  5.966 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  7.940 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 15.245 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  5.380 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  5.812 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 11.785 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  9.890 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  5.784 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  3.254 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  5.495 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 10.630 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 11.234 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 14.060 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 10.535 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 13.519 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  2.164 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 10.405 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.514 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  9.201 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.129 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:07 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31:41 min
[INFO] Finished at: 2017-12-10T11:55:28+08:00
[INFO] Final Memory: 166M/494M
[INFO] ------------------------------------------------------------------------
[root@rzdatahadoop01 hadoop-2.8.1-src]# 


注：download需要3.5h
不想download就使用.m2.tar.gz -->解压-->root家目录

13编译好的tar包
 /opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

提醒： 
1、有时候编译过程中会出现下载某个包的时间太久，这是由于连接网站的过程中会出现假死，
此时按ctrl+c，重新运行编译命令。 
2、如果出现缺少了某个文件的情况，则要先清理maven(使用命令 mvn clean) 再重新编译。

感谢各位的阅读！关于“CentOS6.5下Hadoop的编译示例”这篇文章就分享到这里了，希望以上内容可以对大家有一定的帮助，让大家可以学到更多知识，如果觉得文章不错，可以把它分享出去让更多的人看到吧！