Installation d'une plateforme Hortonworks 2.5

Préparation des serveurs

A faire sur l'ensemble des serveurs

  • Installation d'une Ubuntu 14 LTS
  • 30 Go d'espace sur /
  • 3 disques de 250 Go dédiés à Hadoop

Mise à jour de l'OS

sudo aptitude update  
sudo aptitude safe-upgrade  
sudo reboot  

Installation des dépendances

sudo aptitude install ntp  
sudo aptitude install openjdk-7-jre-headless  

Configuration NTP

sudo nano /etc/ntp.conf  
server 172.19.1.20  
sudo /etc/init.d/ntp stop  
sudo ntpdate -s 172.19.1.20  
sudo /etc/init.d/ntp start  

Désactivation de HTP

sudo nano /etc/default/grub  
GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=never"  
sudo update-grub  
sudo reboot  

Création du compte ambari

sudo adduser ambari  

Configuration des "limits"

sudo nano /etc/security/limits.d/hadoop.conf  
hadoop hard nofile 10000  
hadoop soft nofile 10000  

Configuration de sudo

sudo visudo  
# Ambari Customizable Users
ambari ALL=(ALL) NOPASSWD:SETENV: /bin/su hdfs *,/bin/su ambari-qa *,/bin/su ranger *,/bin/su zookeeper *,/bin/su knox *,/bin/su falcon *,/bin/su ams *, /bin/su flume *,/bin/su hbase *,/bin/su spark *,/bin/su accumulo *,/bin/su hive *,/bin/su hcat *,/bin/su kafka *,/bin/su mapred *,/bin/su oozie *,/bin/su sqoop *,/bin/su storm *,/bin/su tez *,/bin/su atlas *,/bin/su yarn *,/bin/su kms *  
# Ambari Non-Customizable Users
ambari ALL=(ALL) NOPASSWD:SETENV: /bin/su mysql *  
# Ambari Commands
ambari ALL=(ALL) NOPASSWD:SETENV: /usr/bin/yum,/usr/bin/ambari-python-wrap,/usr/bin/zypper,/usr/bin/apt-get, /bin/mkdir, /usr/bin/test, /bin/ln, /bin/chown, /bin/chmod, /bin/chgrp, /usr/sbin/groupadd, /usr/sbin/groupmod, /usr/sbin/useradd, /usr/sbin/usermod, /bin/cp, /usr/sbin/setenforce, /usr/bin/test, /usr/bin/stat, /bin/mv, /bin/sed, /bin/rm, /bin/kill, /bin/readlink, /usr/bin/pgrep, /bin/cat, /usr/bin/unzip, /bin/tar, /usr/bin/tee, /bin/touch, /usr/bin/hdp-select, /usr/bin/conf-select, /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh, /usr/lib/hadoop/bin/hadoop-daemon.sh, /usr/lib/hadoop/sbin/hadoop-daemon.sh, /sbin/chkconfig gmond off, /sbin/chkconfig gmetad off, /etc/init.d/httpd *, /sbin/service hdp-gmetad start, /sbin/service hdp-gmond start, /usr/sbin/gmond, /usr/sbin/update-rc.d ganglia-monitor *, /usr/sbin/update-rc.d gmetad *, /etc/init.d/apache2 *, /usr/sbin/service hdp-gmond *, /usr/sbin/service hdp-gmetad *, /sbin/service mysqld *, /usr/bin/python2.6 /var/lib/ambari-agent/data/tmp/validateKnoxStatus.py *, /usr/hdp/current/knox-server/bin/knoxcli.sh *, /usr/bin/dpkg *, /bin/rpm *, /usr/sbin/hst *  
# Ambari Ranger Commands
ambari ALL=(ALL) NOPASSWD:SETENV: /usr/hdp/*/ranger-usersync/setup.sh, /usr/bin/ranger-usersync-stop, /usr/bin/ranger-usersync-start, /usr/hdp/*/ranger-admin/setup.sh *, /usr/hdp/*/ranger-knox-plugin/disable-knox-plugin.sh *, /usr/hdp/*/ranger-storm-plugin/disable-storm-plugin.sh *, /usr/hdp/*/ranger-hbase-plugin/disable-hbase-plugin.sh *, /usr/hdp/*/ranger-hdfs-plugin/disable-hdfs-plugin.sh *,  /usr/hdp/current/ranger-admin/ranger_credential_helper.py, /usr/hdp/current/ranger-kms/ranger_credential_helper.py, /usr/hdp/*/ranger-*/ranger_credential_helper.py  

Formater et monter les disques pour HDFS

sudo cfdisk /dev/sdb  
sudo cfdisk /dev/sdc  
sudo cfdisk /dev/sdd

sudo mkfs.ext4 -m 0 /dev/sdb1  
sudo mkfs.ext4 -m 0 /dev/sdc1  
sudo mkfs.ext4 -m 0 /dev/sdd1

sudo mkdir -p /data1  
sudo mkdir -p /data2  
sudo mkdir -p /data3

sudo nano /etc/fstab  
/dev/sdb1 /data1 ext4 defaults,nofail,noatime 0 2
/dev/sdc1 /data2 ext4 defaults,nofail,noatime 0 2
/dev/sdd1 /data3 ext4 defaults,nofail,noatime 0 2
sudo mount -a  

Configuration de la résolution des noms

sudo nano /etc/hosts  
172.19.2.221 node01  
172.19.2.222 node02  
172.19.2.223 node03  
172.19.2.224 node04  

Vérification du système

python --version  
java -version  
cat /sys/kernel/mm/transparent_hugepage/enabled  
df -Th  
date  
sudo ntpq -p

ping node01  
ping node02  
ping node03  
ping node04  

Python must be 2.7

Java must be 7

HTP must be never

Installation de l'agent Ambari

A faire sur l'ensemble des serveurs sauf si vous dédiez un serveur pour ambari server.

sudo wget -O- "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xB9733A7A07513CAD" | sudo apt-key add -  
sudo wget -nv http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.4.1.0/ambari.list -O /etc/apt/sources.list.d/ambari.list  
sudo aptitude update

sudo aptitude install ambari-agent  
sudo nano /etc/ambari-agent/conf/ambari-agent.ini  
hostname=node01  
...
run_as_user=ambari  
sudo /etc/init.d/ambari-agent start  

Installation d'ambari serveur

A faire uniquement sur un noeud (dédié à herberger des rôles master ou dédié à ambari serveur).

sudo aptitude install ambari-server  
sudo ambari-server setup  
Customize user account for ambari-server daemon [y/n] (n)? y  
Enter user account for ambari-server daemon (root):ambari  
Checking JDK...  
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 1  
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y  
Enter advanced database configuration [y/n] (n)? n  
sudo chown -R ambari /var/run/ambari-server  
sudo /etc/init.d/ambari-server start  

Déploiement des services

Hadoop HDSF / Zookeeper

Login

Create cluster

Cluster name

Hadoop version

Package URL

Servers provisionning

Servers checkup

Service to deploy

Hadoop master repartition

Hadoop slave repartition

Hadoop generals settings

Hadoop advanced settings

Deploy services

Résolution des problèmes

HDFS

Exécutez les commandes suivantes :

sudo python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=users  
aptitude remove hdp-select  
rm -rf /usr/hdp  
dpkg --configure -a  

Ensuite il faut réinstaller l'agent ambari