Ceph luminous运维指令01
1.获取CRUSH map
# ceph osd getcrushmap -o {crush-file-bin}
2.反编译CRUSH map
# crushtool -d {crush-file-bin} -o {crush-file-txt}
3.编译CRUSH map
# crushtool -c {crush-file-txt} -o {crush-file-bin}
4.导入CRUSH map
# ceph osd setcrushmap -i {crush-file-bin}
5.显示默认配置
# ceph --show-config
6.显示进程当前配置
# ceph daemon client.rgw.gz-open-dw-c43 config show
# ceph daemon osd.1 config show
# ceph mon.gz-open-dw-c43 config show
7.修改线上服务器的配置
# ceph tell 进程 injectargs "--参数=值"
eg: ceph tell osd.214 injectargs "--debug_osd=0/5"
8.查看bucket的shards
# radosgw-admin metadata list
[
"bucket",
"bucket.instance",
"user"
]
查看所有bucket
# radosgw-admin metadata list bucket [ "my-new-bucket", "api_image_test" ]查看bucket 的信息
# radosgw-admin metadata get bucket:api_image_test { "key": "bucket:api_image_test", "ver": { "tag": "_LLvCxW59Cw1cCA97WqaOPIW", "ver": 25 }, "mtime": "2018-01-20 13:46:40.768173Z", "data": { "bucket": { "name": "api_image_test", "marker": "555c6dcf-7320-4b34-8922-5e005dedc130.74924.1", "bucket_id": "555c6dcf-7320-4b34-8922-5e005dedc130.164446.1", "tenant": "", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" } }, "owner": "tupu_aws3", "creation_time": "2018-01-17 04:17:36.345699Z", "linked": "true", "has_bucket_info": "false" } }获取bucket id 并根据 bucket id 从bucket.install中获取真实shards
“bucket_id”: “555c6dcf-7320-4b34-8922-5e005dedc130.164446.1”
# radosgw-admin metadata get bucket.instance:api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164445.1 { "key": "bucket.instance:api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164445.1", "ver": { "tag": "tjx//BY5y1WFfKLkyWH0zeL5", "ver": 4 }, "mtime": "2018-01-20 13:46:40.051678Z", "data": { "bucket_info": { "bucket": { "name": "api_image_test", "marker": "555c6dcf-7320-4b34-8922-5e005dedc130.74924.1", "bucket_id": "555c6dcf-7320-4b34-8922-5e005dedc130.164445.1", "tenant": "", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" } }, "creation_time": "2018-01-17 04:17:36.345699Z", "owner": "tupu_aws3", "flags": 0, "zonegroup": "346e4a1a-ec3d-45d8-b302-964c9355f11c", "placement_rule": "default-placement", "has_instance_obj": "true", "quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "num_shards": 512, "bi_shard_hash_type": 0, "requester_pays": "false", "has_website": "false", "swift_versioning": "false", "swift_ver_location": "", "index_type": 0, "mdsearch_config": [], "reshard_status": 0, "new_bucket_instance_id": "" }, "attrs": [ { "key": "user.rgw.acl", "val": "AgKrAAAAAwIjAAAACQAAAHR1cHVfYXdzMxIAAABUdXB1IGF3cyB1c2VyIG5vLjEEA3wAAAABAQAAAAkAAAB0dXB1X2F3czMPAAAAAQAAAAkAAAB0dXB1X2F3czMFA0cAAAACAgQAAAAAAAAACQAAAHR1cHVfYXdzMwAAAAAAAAAAAgIEAAAADwAAABIAAABUdXB1IGF3cyB1c2VyIG5vLjEAAAAAAAAAAAAAAAAAAAAA" }, { "key": "user.rgw.idtag", "val": "" } ] } }可以看到shards num 为512
使用rados 查看pool的shard数量
“bucket_id”: “555c6dcf-7320-4b34-8922-5e005dedc130.164446.1”
# rados ls -p default.rgw.buckets.index | grep "555c6dcf-7320-4b34-8922-5e005dedc130.164446" | wc -l 512查看所有的bucket.instance
# radosgw-admin metadata list bucket.instance [ "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164293.2", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164293.1", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164296.1", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.74903.4", "my-new-bucket:555c6dcf-7320-4b34-8922-5e005dedc130.74364.1", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164331.1", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.74903.1", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164445.1", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.74924.2", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.74924.1", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.74903.3", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164446.1", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.74903.2", "api_image_test:555c6dcf-7320-4b34-8922-5e005dedc130.164334.1" ]
9.查看rgw用户信息
查看rgw的用户
# radosgw-admin metadata list user [ "tupu_aws3" ]查看某个用户信息
# radosgw-admin metadata get user:tupu_aws3 { "key": "user:tupu_aws3", "ver": { "tag": "__4y4fY-1_64TPtA2GIiLW8J", "ver": 1 }, "mtime": "2018-01-12 02:12:33.285491Z", "data": { "user_id": "tupu_aws3", "display_name": "Tupu aws user no.1", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "tupu_aws3", "access_key": "OEZQW11EPM4XGXE5Q8DI", "secret_key": "Px5Aq09crqIZ2iqbLP83lrcmtDHWseni8RvH2tDr" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "attrs": [ { "key": "user.rgw.idtag", "val": "" } ] } }
10.查看object的存储节点
查看所有的pool
# ceph osd lspools 3 cephfs2_metadata, 4 cephfs2_data, 7 .rgw.root, 8 default.rgw.control, 9 default.rgw.meta, 10 default.rgw.log, 11 cephfs_metadata, 12 cephfs_data, 13 default.rgw.buckets.index, 14 default.rgw.buckets.data,查看pool的某个object存储位置
# ceph osd map default.rgw.buckets.index "lion.png" osdmap e581 pool 'default.rgw.buckets.index' (13) object 'lion.png' -> pg 13.46eb2e7e (13.7e) -> up ([8,22], p8) acting ([8,22], p8)pg 13.46eb2e7e (13.7e) -> up ([8,22], p8), 13 为poolid，46eb2e7是pgid ，up [8,22]说明分布在 id 为8，22的osd上
11.检测bucket index 是否需要resharding（查看bucket object数量和shards数量）
```
# sudo radosgw-admin bucket limit check
[
{
"user_id": "tupu_aws3",
"buckets": [
{
"bucket": "api_image_test",
"tenant": "",
"num_objects": 51243241,
"num_shards": 512,
"objects_per_shard": 100084,
"fill_status": "OK"
},
{
"bucket": "my-new-bucket",
"tenant": "",
"num_objects": 9520,
"num_shards": 0,
"objects_per_shard": 9520,
"fill_status": "OK"
}
]
}
]
```
12.ceph 在线修改mon，osd，mgr，rgw等参数
1.使用 ceph tell
# ceph tell mon.* injectargs '--mon_max_pg_per_osd 300'2.使用 ceph daemon
获取参数： # ceph daemon client.rgw.gz-open-dw-c43 config show 或者： # ceph daemon /var/run/ceph/ceph-client.rgw.gz-open-dw-c43.asok config show # ceph daemon /var/run/ceph/ceph-client.rgw.gz-open-dw-c43.asok config get 参数名 修改参数： # ceph daemon /var/run/ceph/ceph-client.rgw.gz-open-dw-c43.asok config set 参数名
13.osd关闭 scurb 和 deep scrub
# ceph osd set nodeep-scrub
# ceph osd set noscrub
目前没有配置文件选项可以关闭，只能通过命令关闭，关闭会引起health warn
14.在osd 需要更换的时候设置noout,防止数据迁移
# ceph osd set noout
15.查看当前default.rgw.buckets.index 中存在的分片的数量（包含历史的分片）。
osd 在进行每一次sharing的时候都会保留上一份分片的index数据，这就是为什么radosgw-admin metadata list bucket.instance 会出现同个bucket有多个记录，只有一个是当前使用的instance。其他的都是历史使用，但是任然占据着空间
查看index 下保留的shard数量
# rados ls -p default.rgw.buckets.index .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164331.1.143 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164293.2.112 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164470.1.371 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164470.1.437 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164470.1.974 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.769 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164470.1.513 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164446.1.198 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164470.1.622 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164470.1.650 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164445.1.64 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164470.1.776 ... ...目前这些都是分片数量，有当前使用的，有历史使用但是不需要的。
查看bucket : api_image_test 当前的shard分片,首先找到 当前bucket id
# radosgw-admin metadata get bucket:api_image_test "bucket": { "name": "api_image_test", "marker": "555c6dcf-7320-4b34-8922-5e005dedc130.74924.1", "bucket_id": "555c6dcf-7320-4b34-8922-5e005dedc130.164465.1", "tenant": "", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" } },bucket id ： 555c6dcf-7320-4b34-8922-5e005dedc130.164465.1
找到bucket id后查询具体的shard编号，如下
# rados ls -p default.rgw.buckets.index | grep "555c6dcf-7320-4b34-8922-5e005dedc130.164465.1" .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.475 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.9 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.289 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.510 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.658 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.615 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.148 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.436 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.746 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.265 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.967 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.120 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.380 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.730 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.285 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.157 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.693 .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.769 ... ...rados ls -p default.rgw.buckets.index | grep "555c6dcf-7320-4b34-8922-5e005dedc130.164465.1" | wc -l 1024可以看到当前的是有 1024个在使用的shards，每个key在其对应的dir/bucket下都会占有200B左右的空间。当dir/bucket下面的key数量 很多时，这将使得dir对象很大。不仅包含该dir对象的osd会使用很多内存，而且当dir 对象迁移时所有对该对象的写操作都会锁定
查看当前shard 存在的index数据
# rados -p default.rgw.buckets.index listomapkeys .dir.555c6dcf-7320-4b34-8922-5e005dedc130.164465.1.769 /world/data-node-n5/sd9/2018-01-21/23/582430e17e54319d3c2e7283/15165477251550.09999773679434498.jpg /world/data-node-n5/sd9/2018-01-21/23/582430e17e54319d3c2e7283/15165479245570.7355140441827839.jpg /world/data-node-n5/sd9/2018-01-21/23/582430e17e54319d3c2e7283/15165483283670.17346996187799846.jpg /world/data-node-n5/sd9/2018-01-21/23/582430e17e54319d3c2e7283/15165487280380.09802191084124368.jpg /world/data-node-n5/sd9/2018-01-21/23/584781f9169ad54e5406a1ad/15165500584440.6087021690995902.jpg /world/data-node-n5/sd9/2018-01-21/23/5864f340ae25f35f1bfa8115/15165472634080.4647687004892407.jpg /world/data-node-n5/sd9/2018-01-21/23/5864f340ae25f35f1bfa8115/15165473644020.05123803556781992.jpg /world/data-node-n5/sd9/2018-01-21/23/5864f340ae25f35f1bfa8115/15165479646180.3335829273449882.jpg /world/data-node-n5/sd9/2018-01-21/23/591a7abfc111fa6050d1e6e5/15165502390340.39548784405197224.jpg /world/data-node-n5/sd9/2018-01-21/23/5a0ff1c34422d608d00e9f15/15165487791560.13574221079852333.jpg ... ...一个shard默认存储100000个index
16.radosgw-admin 删除一个bucket（清空数据）
查看需要删除的bucket
# radosgw-admin buckets list [ "api_image_2018-01-30", "my-new-bucket", "api_image_2018-01-31", "api_image_2018-01-26", "api_image_test_2018-01-20", "api_image_2018-01-29", "api_image_2018-01-25", "api_image_2018-01-20", "api_image_2018-01-28", "api_image_test", "my-new-bucket-2", "api_image_2018-01-27", "api_image_2018-01-22", "api_image_test_2018-01-23", "api_image_2018-01-23", "api_image_test_2018-01-22" ]删除bucket
# radosgw-admin bucket rm --bucket=my-new-bucket --purge-objects查询删除速度
# while true; do num1=`radosgw-admin bucket stats --bucket=my-new-bucket | grep num_objects | cut -d " " -f 14`; sleep 60;num2=`radosgw-admin bucket stats --bucket=my-new-bucket | grep num_objects | cut -d " " -f 14`; diff=`expr $num1 - $num2`; echo $diff; done这是个单线程删除过程，会非常慢，大概1分钟1000多个
17.获取bucket的所有object key
# radosgw-admin bi list --bucket=api_image_test_2018-01-20 | grep '"name"' | cut -d ":" -f2 | cut -d '"' -f 2 >> /home/ceph/api_image_test_2018-01-20.keys