{"_id":"56620ebbb401c70d00dde7aa","__v":27,"user":"54d4ec36f6c48a0d00f0f040","version":{"_id":"563a2dba1846790d00895309","__v":3,"project":"54d4ecb5f6c48a0d00f0f041","createdAt":"2015-11-04T16:09:30.844Z","releaseDate":"2015-11-04T16:09:30.844Z","categories":["563a2dbb1846790d0089530a","563a2dbb1846790d0089530b","563a2dbb1846790d0089530c","56620e60f183880d004d3217","5702e5b8f2d6f336005e9025"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"No Mashape","version_clean":"1.1.0","version":"1.1"},"category":{"_id":"56620e60f183880d004d3217","version":"563a2dba1846790d00895309","__v":2,"pages":["56620ebbb401c70d00dde7aa","566226ed0299ea0d008f2cee"],"project":"54d4ecb5f6c48a0d00f0f041","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-12-04T22:06:24.370Z","from_sync":false,"order":1,"slug":"counter-sharding","title":"Counter Sharding Basics"},"project":"54d4ecb5f6c48a0d00f0f041","updates":[],"next":{"pages":[],"description":""},"createdAt":"2015-12-04T22:07:55.828Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"settings":"","results":{"codes":[]},"auth":"required","params":[],"url":""},"isReference":false,"order":0,"body":"A **sharded counter** is counter whose count is stored across multiple, independent data [shards](https://en.wikipedia.org/wiki/Shard_(database_architecture) to enable high throughput counter operations (think: increments and decrements) while maintaining a high degree of data-persistence availability and redundancy.\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Simplistic Counting\"\n}\n[/block]\nIn a simple architecture (i.e., one that doesn't use *sharding*), a counter might be stored in a single row in a datastore, for example a SQL database.  When an increment or decrement occurs, the database row for the counter is locked to prevent simultaneous counter operations from colliding with each other.  Inside the database, the counter is incremented, and the database row is unlocked.  \n\nIn this way, a counter might be incremented from 1 to 2 in a consistent fashion, like the diagram below.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/dM8JNk3JRfKlLXkGxATm_simple-counter.svg\",\n        \"simple-counter.svg\",\n        \"0\",\n        \"0\",\n        \"#c4c4c4\",\n        \"\"\n      ],\n      \"caption\": \"Simple counter increment using a transactional datastore like a SQL database.\"\n    }\n  ]\n}\n[/block]\nOne major drawback with this simplistic approach is poor performance under high loads.  Notice that for each increment or decrement operation, the entire counter must be locked for the duration of the operation.  Under heavy load, this would produce unacceptable bottlenecks for consumers of the counter-service because each increment/decrement must happen in a serial fashion.  For example, if an increment takes 100ms to effect, then 1000 requests to increment the same counter at the same time would take 100 seconds, or nearly two minutes!!\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Enter Counter Shards\"\n}\n[/block]\nUnlike our simplistic counter above, a Sharded Counter overcomes these limitations by breaking the total count into shards, and distributing operations across the shards in a random fashion.  If a client wants to to increment the counter, the Counter Service picks one of the shards at random and increments it.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/KYts196Si6TMUG87EvXg_increment-counter2.svg\",\n        \"increment-counter2.svg\",\n        \"0\",\n        \"0\",\n        \"#c2c2c2\",\n        \"\"\n      ],\n      \"caption\": \"A Sharded Counter increment operation.\"\n    }\n  ]\n}\n[/block]\nNotice that the other shards that were not incremented, but were left untouched.  This means that during the increment operation, these other shards were available to service other requests, either in parallel or at a future point in time.\n\nIf a client wants to know the total count, the Counter Service can read all of the counter shards and sum up their individual counts, like this:\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/53QZu5YdSxOJe4edGFMo_get-count.svg\",\n        \"get-count.svg\",\n        \"0\",\n        \"0\",\n        \"#c1c1c1\",\n        \"\"\n      ],\n      \"caption\": \"Retrieving the total count of a Sharded Counter by parallel querying each counter shard and summing the counts.\"\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Can Sharded Counters Scale?\"\n}\n[/block]\nThey certainly can!  To read more on this topic, checkout the [next section on scalability](/docs/how-scalable-is-a-sharded-counter).","excerpt":"Background and theory behind a Sharded Counter","slug":"whats-a-sharded-counter","type":"basic","title":"What's a Sharded Counter?"}

What's a Sharded Counter?

Background and theory behind a Sharded Counter

A **sharded counter** is counter whose count is stored across multiple, independent data [shards](https://en.wikipedia.org/wiki/Shard_(database_architecture) to enable high throughput counter operations (think: increments and decrements) while maintaining a high degree of data-persistence availability and redundancy. [block:api-header] { "type": "basic", "title": "Simplistic Counting" } [/block] In a simple architecture (i.e., one that doesn't use *sharding*), a counter might be stored in a single row in a datastore, for example a SQL database. When an increment or decrement occurs, the database row for the counter is locked to prevent simultaneous counter operations from colliding with each other. Inside the database, the counter is incremented, and the database row is unlocked. In this way, a counter might be incremented from 1 to 2 in a consistent fashion, like the diagram below. [block:image] { "images": [ { "image": [ "https://files.readme.io/dM8JNk3JRfKlLXkGxATm_simple-counter.svg", "simple-counter.svg", "0", "0", "#c4c4c4", "" ], "caption": "Simple counter increment using a transactional datastore like a SQL database." } ] } [/block] One major drawback with this simplistic approach is poor performance under high loads. Notice that for each increment or decrement operation, the entire counter must be locked for the duration of the operation. Under heavy load, this would produce unacceptable bottlenecks for consumers of the counter-service because each increment/decrement must happen in a serial fashion. For example, if an increment takes 100ms to effect, then 1000 requests to increment the same counter at the same time would take 100 seconds, or nearly two minutes!! [block:api-header] { "type": "basic", "title": "Enter Counter Shards" } [/block] Unlike our simplistic counter above, a Sharded Counter overcomes these limitations by breaking the total count into shards, and distributing operations across the shards in a random fashion. If a client wants to to increment the counter, the Counter Service picks one of the shards at random and increments it. [block:image] { "images": [ { "image": [ "https://files.readme.io/KYts196Si6TMUG87EvXg_increment-counter2.svg", "increment-counter2.svg", "0", "0", "#c2c2c2", "" ], "caption": "A Sharded Counter increment operation." } ] } [/block] Notice that the other shards that were not incremented, but were left untouched. This means that during the increment operation, these other shards were available to service other requests, either in parallel or at a future point in time. If a client wants to know the total count, the Counter Service can read all of the counter shards and sum up their individual counts, like this: [block:image] { "images": [ { "image": [ "https://files.readme.io/53QZu5YdSxOJe4edGFMo_get-count.svg", "get-count.svg", "0", "0", "#c1c1c1", "" ], "caption": "Retrieving the total count of a Sharded Counter by parallel querying each counter shard and summing the counts." } ] } [/block] [block:api-header] { "type": "basic", "title": "Can Sharded Counters Scale?" } [/block] They certainly can! To read more on this topic, checkout the [next section on scalability](/docs/how-scalable-is-a-sharded-counter).