搜索
首页数据库mysql教程New Hash-based Sharding Feature in MongoDB 2.4

Lots of MongoDB users enjoy the flexibility of custom shard keys in organizing a sharded collection’s documents. For certain common workloads though, like key/value lookup, using the natural choice of _id as a shard key isn’t optimal bec

Lots of MongoDB users enjoy the flexibility of custom shard keys in organizing a sharded collection’s documents. For certain common workloads though, like key/value lookup, using the natural choice of _id as a shard key isn’t optimal because default ObjectId’s are ascending, resulting in poor write distribution. ?Creating randomized _ids or choosing another well-distributed field is always possible, but this adds complexity to an app and is another place where something could go wrong.

To help keep these simple workloads simple, in 2.4 MongoDB added the new Hash-based shard key feature. ?The idea behind Hash-based shard keys is that MongoDB will do the work to randomize data distribution for you, based on whatever kind of document identifier you like. ?So long as the identifier has a high cardinality, the documents in your collection will be spread evenly across the shards of your cluster. ?For heavy workloads with lots of individual document writes or reads (e.g. key/value), this is usually the best choice. ?For workloads where getting ranges of documents is more important (i.e. find recent documents from all users), other choices of shard key may be better suited.

Hash-based sharding in an existing collection

To start off with Hash-based sharding, you need the name of the collection you’d like to shard and the name of the hashed “identifier" field for the documents in the collection. ?For example, we might want to create a sharded “mydb.webcrawler" collection, where each document is usually found by a “url" field. ?We can populate the collection with sample data using:

shell$ wget http://en.wikipedia.org/wiki/Web_crawler -O web_crawler.html
shell$ mongo 
connecting to: /test
> use mydb
switched to db mydb
> cat("web_crawler.html").split("\n").forEach( function(line){
... var regex = /a href="http://blog.mongodb.org/post/\""([^\"]*)\"/; if (regex.test(line)) { db.webcrawler.insert({ "url" : regex.exec(line)[1] }); }})
> db.webcrawler.find()
...
{ "_id" : ObjectId("5162fba3ad5a8e56b7b36020"), "url" : "/wiki/OWASP" }
{ "_id" : ObjectId("5162fba3ad5a8e56b7b3603d"), "url" : "/wiki/Image_retrieval" }
{ "_id" : ObjectId("5162fba3ad5a8e56b7b3603e"), "url" : "/wiki/Video_search_engine" }
{ "_id" : ObjectId("5162fba3ad5a8e56b7b3603f"), "url" : "/wiki/Enterprise_search" }
{ "_id" : ObjectId("5162fba3ad5a8e56b7b36040"), "url" : "/wiki/Semantic_search" }
...

Just for this example, we multiply this data ~x2000 (otherwise we won’t get any pre-splitting in the collection because it’s too small):

> for (var i = 0; i 
<p><span>Next, we create a hashed index on this field:</span></p>
<pre class="brush:php;toolbar:false">> db.webcrawler.ensureIndex({ url : "hashed" })

As usual, the creation of the hashed index doesn’t prevent other types of indices from being created as well.

Then we shard the “mydb.webcrawler" collection using the same field as a Hash-based shard key:

> db.printShardingStatus(true)
--- Sharding Status ---
sharding version: {
  "_id" : 1,
  "version" : 3,
  "minCompatibleVersion" : 3,
  "currentVersion" : 4,
  "clusterId" : ObjectId("5163032a622c051263c7b8ce")
}
shards:
  {  "_id" : "test-rs0",  "host" : "test-rs0/nuwen:31100,nuwen:31101" }
  {  "_id" : "test-rs1",  "host" : "test-rs1/nuwen:31200,nuwen:31201" }
  {  "_id" : "test-rs2",  "host" : "test-rs2/nuwen:31300,nuwen:31301" }
  {  "_id" : "test-rs3",  "host" : "test-rs3/nuwen:31400,nuwen:31401" }
databases:
  {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
  {  "_id" : "mydb",  "partitioned" : true,  "primary" : "test-rs0" }
      mydb.webcrawler
          shard key: { "url" : "hashed" }
          chunks:
              test-rs0    4
          { "url" : { "$minKey" : 1 } } -->> { "url" : NumberLong("-4837773290201122847") } on : test-rs0 { "t" : 1, "i" : 3 }
          { "url" : NumberLong("-4837773290201122847") } -->> { "url" : NumberLong("-2329535691089872938") } on : test-rs0 { "t" : 1, "i" : 4 }
          { "url" : NumberLong("-2329535691089872938") } -->> { "url" : NumberLong("3244151849123193853") } on : test-rs0 { "t" : 1, "i" : 1 }
          { "url" : NumberLong("3244151849123193853") } -->> { "url" : { "$maxKey" : 1 } } on : test-rs0 { "t" : 1, "i" : 2 }

you can see that the chunk boundaries are 64-bit integers (generated by hashing the “url" field). ?When inserts or queries target particular urls, the query can get routed using the url hash to the correct chunk.

Sharding a new collection

Above we’ve sharded an existing collection, which will result in all the chunks of a collection initially living on the same shard. ?The balancer takes care of moving the chunks around, as usual, until we get an even distribution of data.

Much of the time though, it’s better to shard the collection before we add our data - this way MongoDB doesn’t have to worry about moving around existing data. ?Users of sharded collections are familiar with pre-splitting - where empty chunks can be quickly balanced around a cluster before data is added. ?When sharding a new collection using Hash-based shard keys, MongoDB will take care of the presplitting for you. Similarly sized ranges of the Hash-based key are distributed to each existing shard, which means that no initial balancing is needed (unless of course new shards are added).

Let’s see what happens when we shard a new collection webcrawler_empty the same way:

> sh.stopBalancer()
Waiting for active hosts...
Waiting for the balancer lock...
Waiting again for active hosts after balancer is off...
> db.webcrawler_empty.ensureIndex({ url : "hashed" })
> sh.shardCollection("mydb.webcrawler_empty", { url : "hashed" })
{ "collectionsharded" : "mydb.webcrawler_empty", "ok" : 1 }
> db.printShardingStatus(true)
--- Sharding Status ---
...
      mydb.webcrawler_empty
          shard key: { "url" : "hashed" }
          chunks:
              test-rs0    2
              test-rs1    2
              test-rs2    2
              test-rs3    2
          { "url" : { "$minKey" : 1 } } -->> { "url" : NumberLong("-6917529027641081850") } on : test-rs0 { "t" : 4, "i" : 2 }
          { "url" : NumberLong("-6917529027641081850") } -->> { "url" : NumberLong("-4611686018427387900") } on : test-rs0 { "t" : 4, "i" : 3 }
          { "url" : NumberLong("-4611686018427387900") } -->> { "url" : NumberLong("-2305843009213693950") } on : test-rs1 { "t" : 4, "i" : 4 }
          { "url" : NumberLong("-2305843009213693950") } -->> { "url" : NumberLong(0) } on : test-rs1 { "t" : 4, "i" : 5 }
          { "url" : NumberLong(0) } -->> { "url" : NumberLong("2305843009213693950") } on : test-rs2 { "t" : 4, "i" : 6 }
          { "url" : NumberLong("2305843009213693950") } -->> { "url" : NumberLong("4611686018427387900") } on : test-rs2 { "t" : 4, "i" : 7 }
          { "url" : NumberLong("4611686018427387900") } -->> { "url" : NumberLong("6917529027641081850") } on : test-rs3 { "t" : 4, "i" : 8 }
          { "url" : NumberLong("6917529027641081850") } -->> { "url" : { "$maxKey" : 1 } } on : test-rs3 { "t" : 4, "i" : 9 }

As you can see, the new empty collection is already well-distributed and ready to use. ?Be aware though - any balancing currently in progress can interfere with moving the empty initial chunks off the initial shard, balancing will take priority (hence the initial stopBalancer step). Like before, eventually the balancer will distribute all empty chunks anyway, but if you are preparing for a immediate data load it’s probably best to stop the balancer beforehand.

That’s it - you now have a pre-split collection on four shards using Hash-based shard keys. ?Queries and updates on exact urls go to randomized shards and are balanced across the cluster:

> db.webcrawler_empty.find({ url: "/wiki/OWASP" }).explain()
{
  "clusteredType" : "ParallelSort",
  "shards" : {
      "test-rs2/nuwen:31300,nuwen:31301" : [ ... ]
...

However, the trade-off with Hash-based shard keys is that ranged queries and multi-updates must hit all shards:

> db.webcrawler_empty.find({ url: /^\/wiki\/OWASP/ }).explain()
{
  "clusteredType" : "ParallelSort",
  "shards" : {
      "test-rs0/nuwen:31100,nuwen:31101" : [ ... ],
     "test-rs1/nuwen:31200,nuwen:31201" : [ ... ],
     "test-rs2/nuwen:31300,nuwen:31301" : [ ... ],
     "test-rs3/nuwen:31400,nuwen:31401" : [ ... ]
...

Manual chunk assignment and other caveats

The core benefits of the new Hash-based shard keys are:

  • Easy setup of randomized shard key

  • Automated pre-splitting of empty collections

  • Better distribution of chunks on shards for isolated document writes and reads

The standard split and moveChunk functions do work with Hash-based shard keys, so it’s still possible to balance your collection’s chunks in any way you like. ?However, the usual “find” mechanism used to select chunks can behave a bit unexpectedly since the specifier is a document which is hashed to get the containing chunk. ?To keep things simple, just use the new “bounds” parameter when manually manipulating chunks of hashed collections (or all collections, if you prefer):

> use admin
> db.runCommand({ split : "mydb.webcrawler_empty", bounds : [{ "url" : NumberLong("2305843009213693950") }, { "url" : NumberLong("4611686018427387900") }] })
> db.runCommand({ moveChunk : "mydb.webcrawler_empty", bounds : [{ "url" : NumberLong("2305843009213693950") }, { "url" : NumberLong("4611686018427387900") }], to : "test-rs3" })

There are a few other caveats as well - in particular with tag-aware sharding. ?Tag-aware sharding is a feature we released in MongoDB 2.2, which allows you to attach labels to a subset of shards in a cluster. This is valuable for “pinning" collection data to particular shards (which might be hosted on more powerful hardware, for example). ?You can also tag ranges of a collection differently, such that a collection sharded by { “countryCode" : 1 } would have chunks only on servers in that country.

Hash-based shard keys are compatible with tag-aware sharding. ?As in any sharded collection, you may assign chunks to specific shards, but since the chunk ranges are based on the value of the randomized hash of the shard key instead of the shard key itself, this is usually only useful for tagging the whole range to a specific set of shards:

> sh.addShardTag("test-rs2", "DC1")
sh.addShardTag("test-rs3", "DC1")

The above commands assign a hypothetical data center tag “DC1” to shards -rs2 and -rs3, which could indicate that -rs2 and -rs3 are in a particular location. ?Then, by running:

> sh.addTagRange("mydb.webcrawler_empty", { url : MinKey }, { url : MaxKey }, "DC1" )

we indicate to the cluster that the mydb.webcrawler_empty collection should only be stored on “DC1” shards. ?After letting the balancer work:

> db.printShardingStatus(true)
--- Sharding Status ---
...
      mydb.webcrawler_empty
          shard key: { "url" : "hashed" }
          chunks:
              test-rs2    4
              test-rs3    4
          { "url" : { "$minKey" : 1 } } -->> { "url" : NumberLong("-6917529027641081850") } on : test-rs2 { "t" : 5, "i" : 0 }
          { "url" : NumberLong("-6917529027641081850") } -->> { "url" : NumberLong("-4611686018427387900") } on : test-rs3 { "t" : 6, "i" : 0 }
          { "url" : NumberLong("-4611686018427387900") } -->> { "url" : NumberLong("-2305843009213693950") } on : test-rs2 { "t" : 7, "i" : 0 }
          { "url" : NumberLong("-2305843009213693950") } -->> { "url" : NumberLong(0) } on : test-rs3 { "t" : 8, "i" : 0 }
          { "url" : NumberLong(0) } -->> { "url" : NumberLong("2305843009213693950") } on : test-rs2 { "t" : 4, "i" : 6 }
          { "url" : NumberLong("2305843009213693950") } -->> { "url" : NumberLong("4611686018427387900") } on : test-rs2 { "t" : 4, "i" : 7 }
          { "url" : NumberLong("4611686018427387900") } -->> { "url" : NumberLong("6917529027641081850") } on : test-rs3 { "t" : 4, "i" : 8 }
          { "url" : NumberLong("6917529027641081850") } -->> { "url" : { "$maxKey" : 1 } } on : test-rs3 { "t" : 4, "i" : 9 }
           tag: DC1  { "url" : { "$minKey" : 1 } } -->> { "url" : { "$maxKey" : 1 } }

Again, it doesn’t usually make a lot of sense to tag anything other than the full hashed shard key collection to particular shards - by design, there’s no real way to know or control what data is in what range.

Finally, remember that Hash-based shard keys can (right now) only distribute documents based on the value of a single field. ?So, continuing the example above, it isn’t directly possible to use “url" + “timestamp" as a Hash-based shard key without storing the combination in a single field in your application, for example:

url_and_ts : { url : <url>, timestamp : <timestamp> }</timestamp></url>

The sub-document will be hashed as a unit.

If you’re interested in learning more about Hash-based sharding, register for the Hash-based sharding feature demo on May 2.

声明
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系admin@php.cn
java中的new关键字怎么使用java中的new关键字怎么使用May 03, 2023 pm 10:16 PM

1.概念在Java语言里,“new”表达式是负责创建实例的,其中会调用构造器去对实例做初始化;构造器自身的返回值类型是void,并不是“构造器返回了新创建的对象的引用”,而是new表达式的值是新创建的对象的引用。2.用途新建类的对象3.工作机制为对象成员分配内存空间,并指定默认值对成员变量进行显式初始化执行构造方法计算并返回引用值4.实例new操作往往意味着内存中的开辟新的内存空间,这个内存空间分配在内存中的堆区,受到jvm控制,自动进行内存管理。这里我们就是用String这个类来举例说明。pu

Redis实现数据的分段(Sharding)详解Redis实现数据的分段(Sharding)详解Jun 20, 2023 pm 04:52 PM

Redis是一个高性能的键值对存储系统,它常用于缓存、排行榜等应用场景。当数据量变得越来越大时,单机的Redis可能会遇到性能瓶颈,这时我们可以通过将数据分段存储到多个Redis节点上来实现横向扩展。这就是Redis的数据分段(Sharding)。Redis的数据分段可以通过一下几个步骤完成:设定分片规则首先需要设定分片规则。Redis的分片可以按照key值

js中new操作符工作原理是什么js中new操作符工作原理是什么Feb 19, 2024 am 11:17 AM

js中new操作符工作原理是什么,需要具体代码示例js中的new操作符是用来创建对象的关键字。它的作用是根据指定的构造函数创建一个新的实例对象,并返回该对象的引用。在使用new操作符时,实际上进行了以下几个步骤:创建一个新的空对象;将该空对象的原型指向构造函数的原型对象;将构造函数的作用域赋给新对象(因此this指向了新对象);执行构造函数中的代码,并给新对

go语言中make和new的区别是什么go语言中make和new的区别是什么Jan 09, 2023 am 11:44 AM

区别:1、make只能用来分配及初始化类型为slice、map、chan的数据;而new可以分配任意类型的数据。2、new分配返回的是指针,即类型“*Type”;而make返回引用,即Type。3、new分配的空间会被清零;make分配空间后,会进行初始化。

New Fujifilm fixed-lens GFX camera to debut new medium format sensor, could kick off all-new seriesNew Fujifilm fixed-lens GFX camera to debut new medium format sensor, could kick off all-new seriesSep 27, 2024 am 06:03 AM

Fujifilm has seen a lot of success in recent years, largely due to its film simulations and the popularity of its compact rangefinger-style cameras on social media. However, it doesn't seem to be resting on its laurels, according to Fujirumors. The u

如何使用Java中的clone()方法替代new关键字?如何使用Java中的clone()方法替代new关键字?Apr 22, 2023 pm 07:55 PM

使用clone()代替new在Java中新建对象实例最常用的方法是使用new关键字。JDK对new的支持非常好,使用new关键字创建轻量级对象时,速度非常快。但是,对于重量级对象,由于对象在构造函数中可能会进行一些复杂且耗时的操作,因此,构造函数的执行时间可能会比较长。导致系统短期内无法获得大量的实例。为了解决这个问题,可以使用Object.clone()方法。Object.clone()方法可以绕过构造函数,快速复制一个对象实例。但是,在默认情况下,clone()方法生成的实例只是原对象的浅拷

java中如何使用new实例化java中如何使用new实例化May 16, 2023 pm 07:04 PM

1.概念就是“创建一个Java对象”-----分配内存并返回指向该内存的引用。2.注意事项(1)Java关键字new是一个运算符。与+、-、*、/等运算符具有相同或类似的优先级。(2)创建一个Java对象需要三步:声明引用变量、实例化、初始化对象实例。(3)实列化之前会默认调用父类的无参构造,也就是要创建父类的对象3.两种实例化方法(1)对象名=new类名(参数1,参数2...参数n);对象名.方法();(2)new类名(参数1,参数2...参数n).方法;4.实例用一个简单的代码来说明使对象实

See all articles

热AI工具

Undresser.AI Undress

Undresser.AI Undress

人工智能驱动的应用程序,用于创建逼真的裸体照片

AI Clothes Remover

AI Clothes Remover

用于从照片中去除衣服的在线人工智能工具。

Undress AI Tool

Undress AI Tool

免费脱衣服图片

Clothoff.io

Clothoff.io

AI脱衣机

AI Hentai Generator

AI Hentai Generator

免费生成ai无尽的。

热门文章

R.E.P.O.能量晶体解释及其做什么(黄色晶体)
2 周前By尊渡假赌尊渡假赌尊渡假赌
仓库:如何复兴队友
1 个月前By尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island冒险:如何获得巨型种子
4 周前By尊渡假赌尊渡假赌尊渡假赌

热工具

适用于 Eclipse 的 SAP NetWeaver 服务器适配器

适用于 Eclipse 的 SAP NetWeaver 服务器适配器

将Eclipse与SAP NetWeaver应用服务器集成。

EditPlus 中文破解版

EditPlus 中文破解版

体积小,语法高亮,不支持代码提示功能

Dreamweaver Mac版

Dreamweaver Mac版

视觉化网页开发工具

记事本++7.3.1

记事本++7.3.1

好用且免费的代码编辑器

VSCode Windows 64位 下载

VSCode Windows 64位 下载

微软推出的免费、功能强大的一款IDE编辑器