It is friday, so here at Redglue, we use this day for a geekiness activities, where we “should” only do geeky things 🙂

Today i will try to explain why so often you see an orange number in  Ambari HDFS dashboard (used Hortonworks HDP 2.5 on this demonstration)  on “Under replicated blocks”  metric (red square here).

This usually means that some files are “asking” for a specific number of target replicas that are not present or not being able to get the replica. So the question is, how i know which files are asking for a number of replicas that are not available?

The first option is use hdfs fsck:

[hdfs@sandbox tmp]$ hdfs fsck / -storagepolicies
Connecting to namenode via http://sandbox.hortonworks.com:50070/fsck?ugi=hdfs&storagepolicies=1&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /172.17.0.2 for path / at Fri Mar 17 17:40:18 UTC 2017
..............................................................
/flighdata/airports.dat: Under replicated BP-1464254149-172.17.0.2-1477381671113:blk_1073742620_1807. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).

Here you can see the list of files that are asking for more replicas that the ones they have. On this particular case (“single-node cluster”), only one replica is alive, and the file is asking for 3.

Another way of checking this is using hdfs ls

[hdfs@sandbox tmp]$ hdfs dfs -ls /flighdata/
Found 1 items
-rw-r--r-- 3 admin hdfs 1068028 2017-03-17 12:30 /flighdata/airports.dat


The number you see in bold (3) is the replication factor asked for this specific file.

It is possible to change the replication individually for each file:

[hdfs@sandbox tmp]$ hdfs dfs -setrep -w 1 /flighdata/airports.dat
Replication 1 set: /flighdata/airports.dat
Waiting for /flighdata/airports.dat ... done
[hdfs@sandbox tmp]$ hdfs dfs -ls /flighdata/
Found 1 items
-rw-r--r-- 1 admin hdfs 1068028 2017-03-17 12:30 /flighdata/airports.dat

But, why is this happening. Why only some files have “wrong” replication factor?

First, you need to check dfs.replication in hdfs-site.xml. On HDP 2.5 sandbox should be configured to be 1.
It turns out that if you upload your file via FilesView in Ambari, the file actually gets the replication factor of 3.

Here is the post in Hortonworks community about the issue and we are pursuing the issue with Hortonworks.

https://community.hortonworks.com/questions/89527/hdp-25-sandbox-replication-factor-3-in-ambari-file.html#comment-89552

Happy weekend.