三月 07

原文

NameNode是HDFS文件系统的中心。它保存了所有文件的信息。客户端想访问HDFS中的文件时先和NameNode通讯,NameNode会返回Client存储此文件的DataNode列表。
对于如此重要的一个组件,它居然是单点失败的(Single Point Of Failure)。这也意味着如果NameNode挂掉,即使DataNode还是好好的,整个HDFS也不可用。

现在对Namenode 的HA方案已经完成实施并合并到主干,经过严格的测试后将会在后续版本中发布。

Jitendra Pandey announcement on Hortonworks’s blog提到:

Significant enhancements were completed to make HOT Failover work:

  • Configuration changes for HA
  • Notion of active and standby states were added to the Namenode
  • Client-side redirection
  • Standby processing journal from Active
  • Dual block reports to Active and Standby

其主要原理是将NameNode分为两种角色,Active和Standby,Active就是正在进行服务的NameNode,而Standby又分三种情况

  • Cold Standby:是当Active NN已经挂掉后才起来的,它本身没有保存任何数据,这时候并不会减少恢复时间
  • Warm Standby:是在Active NN挂掉前起来的,其中保存了一部分数据,所以在恢复时只需要恢复没有的数据,减少了恢复时间
  • Hot Standby:它里面保存的数据和Active是完成一样的,可以直接热切换到它上面继续服务

详细介绍请看设计文档:NameNode+HA_v2.pdf

生产系统中对NameNode系统的一些建议:

  • Use a good server with lots of RAM. The more RAM you have, the bigger the file system, or the smaller the block size.
  • Use ECC RAM.
  • On Java6u15 or later, run the server VM with compressed pointers -XX:+UseCompressedOops to cut the JVM heap size down.

  • List more than one name node directory in the configuration, so that multiple copies of the file system meta-data will be stored. As long as the directories are on separate disks, a single disk failure will not corrupt the meta-data.
  • Configure the NameNode to store one set of transaction logs on a separate disk from the image.

  • Configure the NameNode to store another set of transaction logs to a network mounted disk.

  • Monitor the disk space available to the NameNode. If free space is getting low, add more storage.

  • Do not host DataNode, JobTracker or TaskTracker services on the same system.

  • 19 条评论 ("Hadoop Namenode HA方案加入到主干代码库中")

    1. BHW Says:

      Way cool! Some extremely valid points! I appreciate you writing
      this write-up plus the rest of the site is extremely good.

    2. manicure Says:

      Great post. I am dealing with some of these issues as well..

    3. manicure Says:

      Hello! Do you know if they make any plugins to safeguard against hackers?
      I'm kinda paranoid about losing everything I've
      worked hard on. Any recommendations?

    4. manicure Says:

      Thanks for the marvelous posting! I actually enjoyed reading
      it, you are a great author.I will always bookmark your blog and will eventually
      come back someday. I want to encourage you to definitely
      continue your great job, have a nice holiday weekend!

    5. Carter Says:

      This is a very good thing, is your best choice, this is a good thing.
      Carter http://miner1973.yolasite.com/miner1973

    6. Lawerence Says:

      My brother suggested I might like this website. He was entirely right. This post actually made my day. You can not imagine simply how much time I had spent for this information! Thanks!

    7. Brice Says:

      My wife and i have been very joyful that Emmanuel could complete his investigations through your ideas he grabbed in your web page. It’s not at all simplistic to just be freely giving procedures that most people could have been selling. We see we need the blog owner to be grateful to for this. The entire explanations you’ve made, the straightforward web site menu, the friendships you will help to engender – it’s many great, and it’s really making our son in addition to our family know that this idea is satisfying, and that’s wonderfully vital. Many thanks for all!

    8. Benny Says:

      GyVaMs Thanks for sharing, this is a fantastic post.Really thank you! Much obliged.

    9. Aria Says:

      It’s hilarious that you should write this as I have been in the progress of checking for data on this topic. Thanks for the input on the topic.

    10. Andrea Says:

      Woah this weblog is magnificent i like studying your posts. Stay up the great work! You know, many individuals are hunting around for this info, you could aid them greatly.

    11. Nevaeh Says:

      This is really interesting, You are an excessively skilled blogger. I’ve joined your feed and look ahead to looking for more of your great post. Additionally, I have shared your site in my social networks!

    12. Heath Says:

      There’s definately a great deal to learn about this topic. I love all of the points you made.

    13. Jessie Says:

      you’re actually a good webmaster. The website loading velocity is amazing. It sort of feels that you are doing any distinctive trick. Furthermore, The contents are masterwork. you’ve performed a fantastic task in this subject!

    14. Ariana Says:

      If you could message me with a few tips on how you made this website look this cool, Id be appreciative!

    15. Elmer Says:

      Thank you for the sensible critique. Me and my neighbor were just preparing to do a little research about this. We got a grab a book from our area library but I think I learned more from this post. I am very glad to see such wonderful information being shared freely out there.

    16. Aria Says:

      This is very good article, I am very interested in its topic and read them was a pleasure.

    17. Aria Says:

      One might survive distress excluding not disgrace.

    18. Ariana Says:

      This was a great post, thanks for the info.

    19. Lucy Says:

      I am not sure where you are getting your info, but great topic. I needs to spend some time learning more or understanding more. Thanks for great info I was looking for this information for my mission.

    发表评论