Core-site.xml trash interval, proxy set up for ambari running as root,rack awareness script

1.Trash is configured by two properties, fs.trash.checkpoint.interval and fs.trash.interval in core-site.xml.

2.In order to use the Ambari Files View, you need to setup an HDFS proxy user for the Ambari daemon

account.

For example, if the AmbariServer daemon is running as root, you configure a proxy user for root

in the Ambari Web UI by selecting:

Services > HDFS > Configs > Advanced > Custom coresite > Add Property

Add the properties and values(In core-site.xml)

 hadoop.proxyuser.root.groups=* and

hadoop.proxyuser.root.hosts=*

to the file as shown above. Save the changes and restart any affected services as indicated in the Ambari Web UI. This assumes that Ambari was installed and runs as the root user account. If this is not the case,

change root to the appropriate user account.

3. net.topology.script.file.name property Configuring rack awareness

/etc/hadoop/conf/topology_script.py. This script references the mapping file

/etc/hadoop/conf/topology_mapping.data

4. In Ambari when using existing MIT KDC /active directory to map users and groups in unix to Hadoop it uses a rule-based system to create mappings between service principals and their related UNIX usernames. The rules are specified using the configuration property hadoop.security.auth_to_local

=default

WebHDFS information

1) The Java native API libraries use RPC over port 8020 while the WebHDFS REST API uses port 50070 to connect to the NameNode and port 50075 to connect to a DataNode.

2) WebHDFS uses HTTP operations like GET, POST, PUT, and DELETE for file access and administration.

3) WebHDFS is compatible with Kerberos authentication. It uses the Simple and Protected

GSSAPI Negotiation Mechanism (SPNEGO), which extends Kerberos to Web applications.

4) Writing a file is a two-step process.

Create a file by creating a file name on the NameNode:

curl -i -X PUT

"http://<NameNode>:50070/webhdfs/v1/web/mydata/largefile.json?op=CREATE".

The output from this command includes the URL used to write data to the file.

• Write to the file by sending data to the DataNodes:

curl –i –PUT –T largefile.json

“http://<DataNode>:50075/webhdfs/v1/web/mydata/largefile.json?op=CREATE&u

ser.name=root&namenoderpcaddress=node1:8020&overwrite=false”

• The curl command can perform a write operation using a single command that performs both

steps:

curl –I –X PUT largefile.json –L

“http://<NameNode>:50070/webhdfs/v1/web/mydata/largefile.json?op=CREATE&u

ser.name=root"

5) 8. If Kerberos is enabled, WebHDFS requires the configuration of two additional hdfs-site.xml

properties.The property names are

dfs.web.authentication.kerberos.principal=”HTTP:/$<FQDN>@$<REALM_NAME>.com”/” and

dfs.web.authentication.kerberos.keytab.=” /etc/security/spengo.service.keytab“

6) Reading a file named webdata:

curl -i -L

"http://<NameNode>:50070/webhdfs/v1/web/mydata/webdata?op=OPEN&user.name=

jason”

7) Creating a directory named mydata:

curl -i -X PUT

"http://<NameNode>:50070/webhdfs/v1/web/mydata?op=MKDIRS&user.name=jason”

• Listing a directory named mydata:

curl -i

"http://<NameNode>:50070/webhdfs/v1/web/mydata?op=LISTSTATUS&user.name=ja

son”

WebHDFS Authentication

When security is off (Kerberos not enabled), the user that is authenticated is the user set in the

user.name=<name> included in the URL. If user.name is not included in the URL, the server may either set the authenticated user to a default Web user, if there is one, or return an error response.

When security is on (Kerberos is enabled), authentication is performed by either Hadoop delegation token or Kerberos SPNEGO. The user encoded in the delegation=<token> argument is authenticated, or the user is authenticated by SPNEGO.

Hdfs-site.xml configurations

Hdfs-site.xml

1)The number of past edits files to retain is controlled by the

dfs.namenode.num.extra.edits.retained

2.The number of fsimage checkpoint files to retain is controlled by the

dfs.namenode.num.checkpoints.retained.

3. NameNodes persist HDFS storage state information to disk. The value recorded in the

dfs.namenode.name.dir , dfs.namenode.edits.dir

4. dfs.namennode.safemode.threshold-pct4

Minimally replicated means at least one replica is available. This percentage is determined by the

5.failed disks tolerated by HDFS DataNodes?

dfs.datanode.failed.volumes.tolerated

6. The HDFS superuser account is determined by the dfs.cluster.administrators

7. dfs.webhdfs.enabled=”true” To verify that WebHDFS

8. If Kerberos is enabled, WebHDFS requires the configuration of two additional hdfs-site.xml

properties.The property names are

dfs.web.authentication.kerberos.principal=”HTTP:/$<FQDN>@$<REALM_NAME>.com”/” and

dfs.web.authentication.kerberos.keytab.=” /etc/security/spengo.service.keytab“

9. Only the dfs.namenode.acls.enabled property needs to be configured as true to set an ACL. NameNode rejects all attemps if this property is not enabled.

10. The mode parameter is calculated using the value of the fs.permissions.umask-mode property

. The default value is 022. For directories the value of 777- 022 = 755, for files the value 666 - 022 = 644 to produce the mode parameter.

11. The default data block size of 128 megabytes is determined by the dfs.blocksize property

12. dfs.replication property in hdfs-site.xml.

13. dfs.datanode.data.dir determines the parent directory used to store HDFS file data blocks. Could list /hadoop/hdfs/data1, /hadoop/hdfs/data2, and so on, which map to multiple disks.

14. dfs.bytes-per-checksum A checksum is calculated and stored on disk for each 512-byte chunk in a data block.

15. dfs.namenode.checkpoint.period Checkpoints occur every hour based on the value

16. If the number of transactions reaches the value in the

dfs.namenode.checkpoint.txns, then a checkpoint occurs immediately. The default is 1,000,000 transactions.

17. The heartbeat interval is three 3 seconds by default,

dfs.heartbeat.interval (datanode sends heart beats availability to namenode)

18. the DataNode is marked as stale  dfs.namenode.stale.datanode.interval. if exceeded value The minimum possible value is three times the heartbeat interval.(30-sceond threshold)

19. dfs.namenode.avoid.read.stale.datanode,when set to true in HDP by default. A stale DataNode is returned at the end of the list of DataNodes when the NameNode is trying to

satisfy client read requests.

20. dfs.namenode.avoid.write.stale.datanode is set to true in HDP by default. Avoids writing to stale datanode.

21.Stale DataNodes are written to only if the number of stale DataNodes exceeds the ratio determined by dfs.namenode.write.stale.datanode.ratio. In HDP it is set to 1, which means that HDFS may write to a stale DataNode.

22. A NameNode declares a DataNode dead when value exceeded 10 minutes 30 seconds

 2 x dfs.namenode.heartbeat.recheckinterval)+ (10 x dfs.heartbeat.interval).

dfs.namenode.heartbeat.recheckinterval default 10 minutes

dfs.heartbeat.interval = 3 seconds default value

23. each unread block for longer periods  has its checksum verified at least every two weeks. dfs.datanode.scan.period.hours value of 0 is disabled. A value 0f 560 for every two weeks

24. The address and port number of the NameNode UI is determined by the dfs.namenode.httpaddress or or dfs.namenode.https-address

Default port no is 50070

25. The most commonly edited file is hdfs-site.xml.

Others include core-site.xml, hadoop-policy.xml, hdfs-log4j, ssl-client.xml, sslserver.xml.

26.dfs.datanode.balance.bandwidthPerSec The default is set to 6,250,000 bytes per second. consider rebalancing 4-5 nodes at a time rather than balancing all 20 at once and also to preserve network bandwidth for running processes other than rebalancing

27. dfs.hosts.exclude =”/etc/hadoop/conf/dfs.exclude” file with the hostname of the DataNode when an administrator decommissions a DataNode.

28. When an administrator decommissions a NodeManager. This file is defined by the

yarn.resourcemanager.nodes.exclude-path =”/etc/hadoop/conf/yarn.exclude” property in the yarn-site.xml file.

29.For Namenode HA settings

---On current journal nodes on their installation paths in hdfs-site.xml set the property

Dfs.journalnode.edits.dir =”/path/to/edits/info/data” where editlogs are stored in the directory paths.

--Locating journal nodes will be set by property in hdfs-site.xml in

Dfs.namenode.shared.edits.dir “qjournal://jn1:8485;jn2:8485;j3:8485”

--dfs.nameservices =”haclustersetup”(The logical hdfs cluster name points to the two namenodes)

-- dfs.ha.namenode.haclustersetup=”nn1,nn2”(names of namenodes)

--dfs.namenode.http-address.<logical clustername>.<names of nodes>

Ex:dfs.namenode.http-address.<haclustersetup>.<nn1>= “node1:50070”

dfs.namenode.http-address.<haclustersetup>.<nn2>= “node2:50070”

--dfs.namenode.rpc-address.<logical clustername>.<name of node>

Ex:dfs.namenode.rpc-address.<haclustersetup>.<nn1>= “node1:8020”

dfs.namenode.rpc-address.<haclustersetup>.<nn2>= “node2:8020”

-- dfs.ha.fencing.methods(values: shell or sshfence)

-- dfs.client.failover.proxy.provider.mycluster property determines the Java class used by the client to determine which NameNode is currently the Active NameNode.”org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

30.  dfs.blockreport.initialDelay =” 120 seconds”At DataNode startup, a block report is sent to the NameNode after a configurable delay.

31. dfs.blockreport.intervalMsec=” 21600000 milliseconds, or 6 hours.” After initial startup, each DataNode periodically sends an updated block report to the NameNode

32. dfs.blockreport.split.threshold =” 1,000,000 blocks.” below the threshold a single block report that includes every HDFS storage directory is sent to the NameNode. Threshold is exceeded block report spans multiple heartbeats