This was an attempt to free up diskspace. We had massively created indexes to speed up reads. So while investigating the disk usage for mongo database we felt that the we can give up some indexes which were not in use. I was running mongo version 2.0.2. Here is how i investigated and completed the task.
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 5.1G 2.8G 65% /
tmpfs 7.4G 0 7.4G 0% /dev/shm
/dev/md0 512G 508G 3.8G 100% /data
I Saw 3.8 GB is was left on the /data. Verified /data to be the /data for the mongo.
Mongo was running version 2.0.2
/etc/mongo.conf showed the mongo was running in the replica set configuration.
So is this server master ?? Lets check it out
$ mongo -u XXXX -p XXXXX admin
MongoDB shell version: 2.0.2
connecting to: admin
SECONDARY> db.getCollectionNames()
Tue Jun 11 09:24:08 uncaught exception: error: { "$err" : "not master and slaveok=false", "code" : 13435 }
So this is not master. Switch to Slave mode.
Alternatively You Can refer the mongo REST webapi at port 28017 to know which is primary.
SECONDARY> rs.slaveOk()
SECONDARY>
{
"databases" : [
{
"name" : "local",
"sizeOnDisk" : 28121759744,
"empty" : false
},
{
"name" : "admin",
"sizeOnDisk" : 218103808,
"empty" : false
},
{
"name" : "hindenberg",
"sizeOnDisk" : 515095134208,
"empty" : false
},
{
"name" : "test",
"sizeOnDisk" : 218103808,
"empty" : false
},
{
"name" : "config",
"sizeOnDisk" : 1,
"empty" : true
}
],
"totalSize" : 543653101568,
"ok" : 1
}
SizeOnDisk is in bytes. I am Seeing The databases local and hindenberg are taking 27G and 480G Approximately.
http://docs.mongodb.org/manual/reference/local-database/ says that it is used to save the replication data and inhouse stuff. Checking the size of oplog for replication data.
SECONDARY> rs.slaveOk()
SECONDARY> use local
switched to db local
SECONDARY> db.oplog.rs.dataSize()
25940727652
SECONDARY>
This says that the size of oplog is 24G. We can see the most of the local database is majorly taken up by the oplog. We get past this.
We need to check the other database "hindenberg"
SECONDARY> db.stats()
{
"db" : "hindenberg",
"collections" : 16,
"objects" : 443885912,
"avgObjSize" : 543.3192339837088,
"dataSize" : 241171753684,
"storageSize" : 254077221152,
"numExtents" : 223,
"indexes" : 20,
"indexSize" : 243503445136,
"fileSize" : 515078356992,
"nsSizeMB" : 16,
"ok" : 1
}
We see that the hindenberg database is taking 236G of data and 226G worth of indexes over this data adding it to 462GB data. We need to see what indexes exist and what amount of space does it occupy on each collection
SECONDARY> db.printCollectionStats()
ghostrider
{
"ns" : "hindenberg.ghostrider",
"count" : 15605608,
"size" : 1428461880,
"avgObjSize" : 91.53516351301404,
"storageSize" : 3965091840,
"numExtents" : 30,
"nindexes" : 1,
"lastExtentSize" : 667406336,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 633746288,
"indexSizes" : {
"_id_" : 633746288
},
"ok" : 1
}
---
random
{
"ns" : "hindenberg.random",
"count" : 75,
"size" : 35760,
"avgObjSize" : 476.8,
"storageSize" : 40960,
"numExtents" : 2,
"nindexes" : 1,
"lastExtentSize" : 32768,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
bbws
{
"ns" : "hindenberg.bbws",
"count" : 20586,
"size" : 1714552,
"avgObjSize" : 83.2872826192558,
"storageSize" : 2793472,
"numExtents" : 5,
"nindexes" : 1,
"lastExtentSize" : 2097152,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 670432,
"indexSizes" : {
"_id_" : 670432
},
"ok" : 1
}
---
jiggs
{
"ns" : "hindenberg.jiggs",
"count" : 2700687,
"size" : 832824148,
"avgObjSize" : 308.3749238619655,
"storageSize" : 986931200,
"numExtents" : 20,
"nindexes" : 4,
"lastExtentSize" : 175112192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 843084592,
"indexSizes" : {
"_id_" : 85692656,
"sOn" : 337644272,
"smId" : 205029552,
"fid" : 214718112
},
"ok" : 1
}
---
stfu
{
"ns" : "hindenberg.stfu",
"count" : 28920,
"size" : 4862320,
"avgObjSize" : 168.13001383125865,
"storageSize" : 11182080,
"numExtents" : 6,
"nindexes" : 2,
"lastExtentSize" : 8388608,
"paddingFactor" : 1.0099999999981542,
"flags" : 1,
"totalIndexSize" : 4194288,
"indexSizes" : {
"_id_" : 997472,
"gmem" : 3196816
},
"ok" : 1
}
---
lms
{
"ns" : "hindenberg.lms",
"count" : 1,
"size" : 64,
"avgObjSize" : 64,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
yolos
{
"ns" : "hindenberg.yolos",
"count" : 2,
"size" : 144,
"avgObjSize" : 72,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
mofos
{
"ns" : "hindenberg.mofos",
"count" : 416702947,
"size" : 235652429060,
"avgObjSize" : 565.5165886311814,
"storageSize" : 243732003136,
"numExtents" : 144,
"nindexes" : 4,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1.4299999977435327,
"flags" : 1,
"totalIndexSize" : 242065139568,
"indexSizes" : {
"_id_" : 19739627488,
"smId" : 34082947808,
"cpair_son" : 78027059152,
"pson" : 110215505120
},
"ok" : 1
}
---
migrations
{
"ns" : "hindenberg.migrations",
"count" : 12,
"size" : 1220,
"avgObjSize" : 101.66666666666667,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
system.indexes
{
"ns" : "hindenberg.system.indexes",
"count" : 20,
"size" : 1912,
"avgObjSize" : 95.6,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 0,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {
},
"ok" : 1
}
---
system.js
{
"ns" : "hindenberg.system.js",
"count" : 2,
"size" : 1316,
"avgObjSize" : 658,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
system.profile
{
"ns" : "hindenberg.system.profile",
"count" : 8901888,
"size" : 3293505232,
"avgObjSize" : 369.9782823598769,
"storageSize" : 5368713184,
"numExtents" : 3,
"nindexes" : 0,
"lastExtentSize" : 1075859456,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {
},
"capped" : 1,
"max" : 2147483647,
"ok" : 1
}
---
system.users
{
"ns" : "hindenberg.system.users",
"count" : 2,
"size" : 196,
"avgObjSize" : 98,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
ufo_preferences
{
"ns" : "hindenberg.ufo_preferences",
"count" : 1000,
"size" : 71520,
"avgObjSize" : 71.52,
"storageSize" : 1003520,
"numExtents" : 1,
"nindexes" : 0,
"lastExtentSize" : 1003520,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {
},
"capped" : 1,
"max" : 1000,
"ok" : 1
}
---
ufo_details
{
"ns" : "hindenberg.ufo_details",
"count" : 208,
"size" : 3039296,
"avgObjSize" : 14612,
"storageSize" : 9408512,
"numExtents" : 5,
"nindexes" : 2,
"lastExtentSize" : 7077888,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 32704,
"indexSizes" : {
"_id_" : 8176,
"u_1" : 24528
},
"ok" : 1
}
---
Looking at the totalIndexSize Parameter in the JSON Objects we see that the the collection "mofos" has the largest index-size. Listing the Indexes.
SECONDARY > db.mofos.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "hindenberg.mofos",
"name" : "_id_"
},
{
"v" : 1,
"key" : {
"smId" : 1
},
"unique" : true,
"ns" : "hindenberg.mofos",
"name" : "smId_1",
"dropDups" : true
},
{
"v" : 1,
"key" : {
"cPair" : 1,
"sOn" : -1,
"_id" : -1
},
"ns" : "hindenberg.mofos",
"name" : "cpair_son"
},
{
"v" : 1,
"key" : {
"p" : 1,
"sOn" : -1,
"_id" : -1
},
"ns" : "hindenberg.mofos",
"name" : "p_Son"
}
]
SECONDARY> db.mofos.dropIndex("conversationPair_1_sentOn_-1__id_-1");
{ "errmsg" : "not master", "ok" : 0 }
Seems The Indexes were not deleted. REASON: Please read through.
Indexes can be created on the servers of replica set independently. And can be deleted independently too. People choose this to tweak application behavior. For example Slave Servers will have more reads => so create more indexes on slave servers.
Here is a caveat,
Some servers are put into clusters from the start. And the indexes are created later when we see data volume leading to slow queries. Once we create an index in primary get written in the oplog and are also transferred to the secondary servers during replication.
So the Indexes which were created by oplog play will think that the indexes are created on primary. This is a bug in mongo, which i think should be taken care of in the later versions of mongo.
SO HOW TO FIX IT:
-----------------------------
Once we take a server out of a replica set, it thinks of everything being a part of independent self. That will be a nice time to hit mongo with some admin level commands ignoring quorum behavior. And when the space is released, we add it back to the replica_set
So i changed the parameters in /etc/mongod.conf to comment out replSet configuration and start mongod process to bind to some other port say 37017, so that the primary is not able to find this secondary slave.
#replSet = hindenberg_replica_set
$ sudo service mongod start
Starting mongod: [ OK ]
forked process: 5048
all output going to: /var/log/mongo/mongod.log
[~]$ tail -f /var/log/mongo/mongod.log
Tue Jun 11 13:56:23 [initandlisten] MongoDB starting : pid=5048 port=37017 dbpath=/data/ 64-bit host=mongohost
Tue Jun 11 13:56:23 [initandlisten] db version v2.0.2, pdfile version 4.5
Tue Jun 11 13:56:23 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Tue Jun 11 13:56:23 [initandlisten] build info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jun 11 13:56:23 [initandlisten] options: { auth: "true", config: "/etc/mongod.conf", dbpath: "/data", directoryperdb: "true", fork: "true", keyFile: "/data/rep.key", logappend: "true", logpath: "/var/log/mongo/mongod.log", port: 37017, rest: "true" }
Tue Jun 11 13:56:23 [initandlisten] journal dir=/data/primary_db/journal
Tue Jun 11 13:56:23 [initandlisten] recover : no journal files present, no recovery needed
Tue Jun 11 13:56:24 [initandlisten] waiting for connections on port 37017
Tue Jun 11 13:56:24 [websvr] admin web console waiting for connections on port 38017
^C
[~]$ sudo netstat -ntupl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:37017 0.0.0.0:* LISTEN 5048/mongod
tcp 0 0 0.0.0.0:38017 0.0.0.0:* LISTEN 5048/mongod
So we see that not mongod process is running on port 37017. Now we will login to the mongo and will try to run the dropIndex again. Note that "SECONDARY" in front of the > prompt is gone showing the service is not running as a replica set mode.
$ mongo -u XXXXX -p XXXXXX admin --port 37017
MongoDB shell version: 2.0.2
connecting to: 127.0.0.1:37017/admin
> use hindenberg
switched to db hindenberg
> db.mofos.dropIndex("cpair_son");
{ "nIndexesWas" : 4, "ok" : 1 }
> db.mofos.dropIndex("p_sOn");
{ "nIndexesWas" : 3, "ok" : 1 }
> db.stats()
{
"db" : "hindenberg",
"collections" : 16,
"objects" : 444191482,
"avgObjSize" : 543.3492968602221,
"dataSize" : 241351129416,
"storageSize" : 254077221152,
"numExtents" : 223,
"indexes" : 18,
"indexSize" : 55336328992,
"fileSize" : 515078356992,
"nsSizeMB" : 16,
"ok" : 1
}
We Now see that the indexSize has reduced to 51GB from 226GB.
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 5.1G 2.8G 65% /
tmpfs 7.4G 0 7.4G 0% /dev/shm
/dev/md0 512G 507G 5.4G 99% /data
We Also See that the disk space is not freed up. REASON: Mongo allocates size before hand. So if a fill of 100MB is made. Mongo will create a fixed file of 2G, fill it with zeroes and will keep on appending until the file is full. It repeats the step everytime a new data is added. This is called Extents. You would also find the number of extents in the db.stats() output. In the above case it is 223. That means hindenberg is actually taking up 223 X 2G that is roughly 446G in data and indexes.
In delete operation it doesnot delete the extents rather reuses it. So we wont be seeing any disk space utilization unntil and unless the virtually freed up space gets used up by mongo.
For this we need to compact the database or repair the database
$ mongo -u XXXXX -p XXXXXX admin --port 37017
MongoDB shell version: 2.0.2
connecting to: 127.0.0.1:37017/admin
> use hisndenberg
switched to db hindenberg
> db.mofos.runCommand("compact")
{ "ok" : 1 }
>
You may want to refer to http://docs.mongodb.org/manual/reference/command/compact/ to additional methods to compact database.
If you have deleted significant amount of indexes from multiple databases and multiple collections, maybe that is when you would want to run a full database repair rather than selecting each database and then collection and then run compact indivudually which you will find in http://docs.mongodb.org/manual/reference/command/repairDatabase/
THE GIVENS
$ df -hFilesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 5.1G 2.8G 65% /
tmpfs 7.4G 0 7.4G 0% /dev/shm
/dev/md0 512G 508G 3.8G 100% /data
I Saw 3.8 GB is was left on the /data. Verified /data to be the /data for the mongo.
Mongo was running version 2.0.2
/etc/mongo.conf showed the mongo was running in the replica set configuration.
So is this server master ?? Lets check it out
$ mongo -u XXXX -p XXXXX admin
MongoDB shell version: 2.0.2
connecting to: admin
SECONDARY> db.getCollectionNames()
Tue Jun 11 09:24:08 uncaught exception: error: { "$err" : "not master and slaveok=false", "code" : 13435 }
So this is not master. Switch to Slave mode.
Alternatively You Can refer the mongo REST webapi at port 28017 to know which is primary.
SECONDARY> rs.slaveOk()
SECONDARY>
START THE DETECTIVE WORK
Lookin at what constituted the data
SECONDARY> db.adminCommand('listDatabases'){
"databases" : [
{
"name" : "local",
"sizeOnDisk" : 28121759744,
"empty" : false
},
{
"name" : "admin",
"sizeOnDisk" : 218103808,
"empty" : false
},
{
"name" : "hindenberg",
"sizeOnDisk" : 515095134208,
"empty" : false
},
{
"name" : "test",
"sizeOnDisk" : 218103808,
"empty" : false
},
{
"name" : "config",
"sizeOnDisk" : 1,
"empty" : true
}
],
"totalSize" : 543653101568,
"ok" : 1
}
SizeOnDisk is in bytes. I am Seeing The databases local and hindenberg are taking 27G and 480G Approximately.
http://docs.mongodb.org/manual/reference/local-database/ says that it is used to save the replication data and inhouse stuff. Checking the size of oplog for replication data.
SECONDARY> rs.slaveOk()
SECONDARY> use local
switched to db local
SECONDARY> db.oplog.rs.dataSize()
25940727652
SECONDARY>
This says that the size of oplog is 24G. We can see the most of the local database is majorly taken up by the oplog. We get past this.
We need to check the other database "hindenberg"
SECONDARY> db.stats()
{
"db" : "hindenberg",
"collections" : 16,
"objects" : 443885912,
"avgObjSize" : 543.3192339837088,
"dataSize" : 241171753684,
"storageSize" : 254077221152,
"numExtents" : 223,
"indexes" : 20,
"indexSize" : 243503445136,
"fileSize" : 515078356992,
"nsSizeMB" : 16,
"ok" : 1
}
We see that the hindenberg database is taking 236G of data and 226G worth of indexes over this data adding it to 462GB data. We need to see what indexes exist and what amount of space does it occupy on each collection
SECONDARY> db.printCollectionStats()
ghostrider
{
"ns" : "hindenberg.ghostrider",
"count" : 15605608,
"size" : 1428461880,
"avgObjSize" : 91.53516351301404,
"storageSize" : 3965091840,
"numExtents" : 30,
"nindexes" : 1,
"lastExtentSize" : 667406336,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 633746288,
"indexSizes" : {
"_id_" : 633746288
},
"ok" : 1
}
---
random
{
"ns" : "hindenberg.random",
"count" : 75,
"size" : 35760,
"avgObjSize" : 476.8,
"storageSize" : 40960,
"numExtents" : 2,
"nindexes" : 1,
"lastExtentSize" : 32768,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
bbws
{
"ns" : "hindenberg.bbws",
"count" : 20586,
"size" : 1714552,
"avgObjSize" : 83.2872826192558,
"storageSize" : 2793472,
"numExtents" : 5,
"nindexes" : 1,
"lastExtentSize" : 2097152,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 670432,
"indexSizes" : {
"_id_" : 670432
},
"ok" : 1
}
---
jiggs
{
"ns" : "hindenberg.jiggs",
"count" : 2700687,
"size" : 832824148,
"avgObjSize" : 308.3749238619655,
"storageSize" : 986931200,
"numExtents" : 20,
"nindexes" : 4,
"lastExtentSize" : 175112192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 843084592,
"indexSizes" : {
"_id_" : 85692656,
"sOn" : 337644272,
"smId" : 205029552,
"fid" : 214718112
},
"ok" : 1
}
---
stfu
{
"ns" : "hindenberg.stfu",
"count" : 28920,
"size" : 4862320,
"avgObjSize" : 168.13001383125865,
"storageSize" : 11182080,
"numExtents" : 6,
"nindexes" : 2,
"lastExtentSize" : 8388608,
"paddingFactor" : 1.0099999999981542,
"flags" : 1,
"totalIndexSize" : 4194288,
"indexSizes" : {
"_id_" : 997472,
"gmem" : 3196816
},
"ok" : 1
}
---
lms
{
"ns" : "hindenberg.lms",
"count" : 1,
"size" : 64,
"avgObjSize" : 64,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
yolos
{
"ns" : "hindenberg.yolos",
"count" : 2,
"size" : 144,
"avgObjSize" : 72,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
mofos
{
"ns" : "hindenberg.mofos",
"count" : 416702947,
"size" : 235652429060,
"avgObjSize" : 565.5165886311814,
"storageSize" : 243732003136,
"numExtents" : 144,
"nindexes" : 4,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1.4299999977435327,
"flags" : 1,
"totalIndexSize" : 242065139568,
"indexSizes" : {
"_id_" : 19739627488,
"smId" : 34082947808,
"cpair_son" : 78027059152,
"pson" : 110215505120
},
"ok" : 1
}
---
migrations
{
"ns" : "hindenberg.migrations",
"count" : 12,
"size" : 1220,
"avgObjSize" : 101.66666666666667,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
system.indexes
{
"ns" : "hindenberg.system.indexes",
"count" : 20,
"size" : 1912,
"avgObjSize" : 95.6,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 0,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {
},
"ok" : 1
}
---
system.js
{
"ns" : "hindenberg.system.js",
"count" : 2,
"size" : 1316,
"avgObjSize" : 658,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
system.profile
{
"ns" : "hindenberg.system.profile",
"count" : 8901888,
"size" : 3293505232,
"avgObjSize" : 369.9782823598769,
"storageSize" : 5368713184,
"numExtents" : 3,
"nindexes" : 0,
"lastExtentSize" : 1075859456,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {
},
"capped" : 1,
"max" : 2147483647,
"ok" : 1
}
---
system.users
{
"ns" : "hindenberg.system.users",
"count" : 2,
"size" : 196,
"avgObjSize" : 98,
"storageSize" : 8192,
"numExtents" : 1,
"nindexes" : 1,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 8176,
"indexSizes" : {
"_id_" : 8176
},
"ok" : 1
}
---
ufo_preferences
{
"ns" : "hindenberg.ufo_preferences",
"count" : 1000,
"size" : 71520,
"avgObjSize" : 71.52,
"storageSize" : 1003520,
"numExtents" : 1,
"nindexes" : 0,
"lastExtentSize" : 1003520,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {
},
"capped" : 1,
"max" : 1000,
"ok" : 1
}
---
ufo_details
{
"ns" : "hindenberg.ufo_details",
"count" : 208,
"size" : 3039296,
"avgObjSize" : 14612,
"storageSize" : 9408512,
"numExtents" : 5,
"nindexes" : 2,
"lastExtentSize" : 7077888,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 32704,
"indexSizes" : {
"_id_" : 8176,
"u_1" : 24528
},
"ok" : 1
}
---
Looking at the totalIndexSize Parameter in the JSON Objects we see that the the collection "mofos" has the largest index-size. Listing the Indexes.
SECONDARY > db.mofos.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "hindenberg.mofos",
"name" : "_id_"
},
{
"v" : 1,
"key" : {
"smId" : 1
},
"unique" : true,
"ns" : "hindenberg.mofos",
"name" : "smId_1",
"dropDups" : true
},
{
"v" : 1,
"key" : {
"cPair" : 1,
"sOn" : -1,
"_id" : -1
},
"ns" : "hindenberg.mofos",
"name" : "cpair_son"
},
{
"v" : 1,
"key" : {
"p" : 1,
"sOn" : -1,
"_id" : -1
},
"ns" : "hindenberg.mofos",
"name" : "p_Son"
}
]
SECONDARY> db.mofos.dropIndex("conversationPair_1_sentOn_-1__id_-1");
{ "errmsg" : "not master", "ok" : 0 }
Seems The Indexes were not deleted. REASON: Please read through.
Indexes can be created on the servers of replica set independently. And can be deleted independently too. People choose this to tweak application behavior. For example Slave Servers will have more reads => so create more indexes on slave servers.
Here is a caveat,
Some servers are put into clusters from the start. And the indexes are created later when we see data volume leading to slow queries. Once we create an index in primary get written in the oplog and are also transferred to the secondary servers during replication.
So the Indexes which were created by oplog play will think that the indexes are created on primary. This is a bug in mongo, which i think should be taken care of in the later versions of mongo.
SO HOW TO FIX IT:
-----------------------------
Once we take a server out of a replica set, it thinks of everything being a part of independent self. That will be a nice time to hit mongo with some admin level commands ignoring quorum behavior. And when the space is released, we add it back to the replica_set
So i changed the parameters in /etc/mongod.conf to comment out replSet configuration and start mongod process to bind to some other port say 37017, so that the primary is not able to find this secondary slave.
#replSet = hindenberg_replica_set
port = 37017
Starting mongod: [ OK ]
forked process: 5048
all output going to: /var/log/mongo/mongod.log
[~]$ tail -f /var/log/mongo/mongod.log
Tue Jun 11 13:56:23 [initandlisten] MongoDB starting : pid=5048 port=37017 dbpath=/data/ 64-bit host=mongohost
Tue Jun 11 13:56:23 [initandlisten] db version v2.0.2, pdfile version 4.5
Tue Jun 11 13:56:23 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Tue Jun 11 13:56:23 [initandlisten] build info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jun 11 13:56:23 [initandlisten] options: { auth: "true", config: "/etc/mongod.conf", dbpath: "/data", directoryperdb: "true", fork: "true", keyFile: "/data/rep.key", logappend: "true", logpath: "/var/log/mongo/mongod.log", port: 37017, rest: "true" }
Tue Jun 11 13:56:23 [initandlisten] journal dir=/data/primary_db/journal
Tue Jun 11 13:56:23 [initandlisten] recover : no journal files present, no recovery needed
Tue Jun 11 13:56:24 [initandlisten] waiting for connections on port 37017
Tue Jun 11 13:56:24 [websvr] admin web console waiting for connections on port 38017
^C
[~]$ sudo netstat -ntupl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:37017 0.0.0.0:* LISTEN 5048/mongod
tcp 0 0 0.0.0.0:38017 0.0.0.0:* LISTEN 5048/mongod
So we see that not mongod process is running on port 37017. Now we will login to the mongo and will try to run the dropIndex again. Note that "SECONDARY" in front of the > prompt is gone showing the service is not running as a replica set mode.
$ mongo -u XXXXX -p XXXXXX admin --port 37017
MongoDB shell version: 2.0.2
connecting to: 127.0.0.1:37017/admin
> use hindenberg
switched to db hindenberg
> db.mofos.dropIndex("cpair_son");
{ "nIndexesWas" : 4, "ok" : 1 }
> db.mofos.dropIndex("p_sOn");
{ "nIndexesWas" : 3, "ok" : 1 }
> db.stats()
{
"db" : "hindenberg",
"collections" : 16,
"objects" : 444191482,
"avgObjSize" : 543.3492968602221,
"dataSize" : 241351129416,
"storageSize" : 254077221152,
"numExtents" : 223,
"indexes" : 18,
"indexSize" : 55336328992,
"fileSize" : 515078356992,
"nsSizeMB" : 16,
"ok" : 1
}
We Now see that the indexSize has reduced to 51GB from 226GB.
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 5.1G 2.8G 65% /
tmpfs 7.4G 0 7.4G 0% /dev/shm
/dev/md0 512G 507G 5.4G 99% /data
We Also See that the disk space is not freed up. REASON: Mongo allocates size before hand. So if a fill of 100MB is made. Mongo will create a fixed file of 2G, fill it with zeroes and will keep on appending until the file is full. It repeats the step everytime a new data is added. This is called Extents. You would also find the number of extents in the db.stats() output. In the above case it is 223. That means hindenberg is actually taking up 223 X 2G that is roughly 446G in data and indexes.
In delete operation it doesnot delete the extents rather reuses it. So we wont be seeing any disk space utilization unntil and unless the virtually freed up space gets used up by mongo.
For this we need to compact the database or repair the database
$ mongo -u XXXXX -p XXXXXX admin --port 37017
MongoDB shell version: 2.0.2
connecting to: 127.0.0.1:37017/admin
> use hisndenberg
switched to db hindenberg
> db.mofos.runCommand("compact")
{ "ok" : 1 }
>
You may want to refer to http://docs.mongodb.org/manual/reference/command/compact/ to additional methods to compact database.
If you have deleted significant amount of indexes from multiple databases and multiple collections, maybe that is when you would want to run a full database repair rather than selecting each database and then collection and then run compact indivudually which you will find in http://docs.mongodb.org/manual/reference/command/repairDatabase/
Anyways that makes about 150G of free space on the partition.
You would also want to find the duplicated in the db indexes and drop them using dropdups command.
Finally, I reverted the changes to the /etc/mongo.conf to make it part of the replica set again and to listen to default port of 27017. The Cluster was happy to accept the new healthy mongo node.
MISSION ACCOMPLISHED. WIN WIN for everyone.
Suggested Reading:
You would also want to find the duplicated in the db indexes and drop them using dropdups command.
Finally, I reverted the changes to the /etc/mongo.conf to make it part of the replica set again and to listen to default port of 27017. The Cluster was happy to accept the new healthy mongo node.
MISSION ACCOMPLISHED. WIN WIN for everyone.
Suggested Reading:
- http://docs.mongodb.org/manual/tutorial/remove-indexes/
- http://www.slideshare.net/mongodb/indexing-with-mongodb
- http://docs.mongodb.org/manual/administration/indexes/
Hope this tutorial helps you enough, Please Let me know