== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:25:48 (1784341548) start mds service on oleg204-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1765 (time 1784341578.19 total 10.00 last 176.49) - open/close/unlink 3423 (time 1784341588.20 total 20.01 last 165.72) - open/close/unlink 5112 (time 1784341598.20 total 30.01 last 168.85) - open/close/unlink 6768 (time 1784341608.20 total 40.01 last 165.59) - open/close/unlink 8443 (time 1784341618.20 total 50.01 last 167.47) - open/close/unlink 10000 (time 1784341627.76 total 59.57 last 162.92) - open/close/unlink 11585 (time 1784341637.76 total 69.57 last 158.45) - open/close/unlink 13223 (time 1784341647.77 total 79.58 last 163.72) - open/close/unlink 14954 (time 1784341657.77 total 89.58 last 173.01) - open/close/unlink 16713 (time 1784341667.77 total 99.58 last 175.87) - open/close/unlink 18459 (time 1784341677.78 total 109.58 last 174.59) - open/close/unlink 20000 (time 1784341686.55 total 118.35 last 175.69) - open/close/unlink 21570 (time 1784341696.55 total 128.36 last 156.92) - open/close/unlink 23324 (time 1784341706.55 total 138.36 last 175.36) - open/close/unlink 25056 (time 1784341716.56 total 148.36 last 173.15) - open/close/unlink 26803 (time 1784341726.56 total 158.37 last 174.62) - open/close/unlink 28529 (time 1784341736.57 total 168.37 last 172.51) - open/close/unlink 30000 (time 1784341745.58 total 177.39 last 163.19) - open/close/unlink 31637 (time 1784341755.58 total 187.39 last 163.65) - open/close/unlink 33221 (time 1784341765.59 total 197.39 last 158.35) - open/close/unlink 34745 (time 1784341775.60 total 207.41 last 152.16) - open/close/unlink 36484 (time 1784341785.60 total 217.41 last 173.89) - open/close/unlink 38227 (time 1784341795.60 total 227.41 last 174.28) - open/close/unlink 40000 (time 1784341805.49 total 237.30 last 179.37) - open/close/unlink 41702 (time 1784341815.49 total 247.30 last 170.17) - open/close/unlink 43456 (time 1784341825.49 total 257.30 last 175.36) - open/close/unlink 45189 (time 1784341835.49 total 267.30 last 173.30) - open/close/unlink 46880 (time 1784341845.50 total 277.30 last 169.03) - open/close/unlink 48551 (time 1784341855.50 total 287.31 last 167.04) - open/close/unlink 50000 (time 1784341864.38 total 296.19 last 163.12) - open/close/unlink 51624 (time 1784341874.39 total 306.19 last 162.36) - open/close/unlink 53276 (time 1784341884.39 total 316.20 last 165.15) - open/close/unlink 54965 (time 1784341894.39 total 326.20 last 168.89) - open/close/unlink 56721 (time 1784341904.39 total 336.20 last 175.58) - open/close/unlink 58226 (time 1784341914.40 total 346.21 last 150.37) - open/close/unlink 59895 (time 1784341924.40 total 356.21 last 166.80) - open/close/unlink 61563 (time 1784341934.41 total 366.22 last 166.70) - open/close/unlink 63235 (time 1784341944.41 total 376.22 last 167.17) total: 64768 open/close/unlink in 384.79 seconds: 168.32 ops/second - open/close/unlink 1769 (time 1784341578.18 total 10.01 last 176.80) - open/close/unlink 3441 (time 1784341588.18 total 20.01 last 167.14) - open/close/unlink 5118 (time 1784341598.19 total 30.01 last 167.62) - open/close/unlink 6766 (time 1784341608.19 total 40.02 last 164.76) - open/close/unlink 8440 (time 1784341618.19 total 50.02 last 167.39) - open/close/unlink 10000 (time 1784341627.84 total 59.66 last 161.70) - open/close/unlink 11587 (time 1784341637.84 total 69.67 last 158.69) - open/close/unlink 13212 (time 1784341647.84 total 79.67 last 162.50) - open/close/unlink 14931 (time 1784341657.84 total 89.67 last 171.89) - open/close/unlink 16676 (time 1784341667.84 total 99.67 last 174.40) - open/close/unlink 18405 (time 1784341677.85 total 109.68 last 172.76) - open/close/unlink 20000 (time 1784341686.74 total 118.57 last 179.44) - open/close/unlink 21556 (time 1784341696.74 total 128.57 last 155.57) - open/close/unlink 23323 (time 1784341706.74 total 138.57 last 176.69) - open/close/unlink 25054 (time 1784341716.74 total 148.57 last 173.08) - open/close/unlink 26793 (time 1784341726.75 total 158.57 last 173.88) - open/close/unlink 28521 (time 1784341736.75 total 168.57 last 172.79) - open/close/unlink 30000 (time 1784341745.83 total 177.66 last 162.75) - open/close/unlink 31609 (time 1784341755.84 total 187.67 last 160.82) - open/close/unlink 33194 (time 1784341765.84 total 197.67 last 158.47) - open/close/unlink 34704 (time 1784341775.85 total 207.68 last 150.87) - open/close/unlink 36422 (time 1784341785.87 total 217.70 last 171.45) - open/close/unlink 38197 (time 1784341795.87 total 227.70 last 177.40) - open/close/unlink 39975 (time 1784341805.88 total 237.71 last 177.71) - open/close/unlink 41685 (time 1784341815.89 total 247.72 last 170.87) - open/close/unlink 43477 (time 1784341825.89 total 257.72 last 179.19) - open/close/unlink 45219 (time 1784341835.89 total 267.72 last 174.12) - open/close/unlink 46893 (time 1784341845.90 total 277.72 last 167.36) - open/close/unlink 48559 (time 1784341855.90 total 287.73 last 166.55) - open/close/unlink 50000 (time 1784341864.68 total 296.51 last 164.15) - open/close/unlink 51624 (time 1784341874.68 total 306.51 last 162.39) - open/close/unlink 53274 (time 1784341884.68 total 316.51 last 164.95) - open/close/unlink 54970 (time 1784341894.68 total 326.51 last 169.58) - open/close/unlink 56688 (time 1784341904.68 total 336.51 last 171.76) - open/close/unlink 58198 (time 1784341914.69 total 346.52 last 150.96) - open/close/unlink 59853 (time 1784341924.69 total 356.52 last 165.44) - open/close/unlink 61547 (time 1784341934.69 total 366.52 last 169.38) - open/close/unlink 63220 (time 1784341944.69 total 376.52 last 167.27) total: 64768 open/close/unlink in 385.50 seconds: 168.01 ops/second - open/close/unlink 1753 (time 1784341578.01 total 10.00 last 175.24) - open/close/unlink 3438 (time 1784341588.01 total 20.00 last 168.49) - open/close/unlink 5130 (time 1784341598.01 total 30.00 last 169.20) - open/close/unlink 6792 (time 1784341608.01 total 40.00 last 166.20) - open/close/unlink 8449 (time 1784341618.02 total 50.01 last 165.63) - open/close/unlink 10000 (time 1784341627.49 total 59.48 last 163.80) - open/close/unlink 11575 (time 1784341637.49 total 69.48 last 157.47) - open/close/unlink 13209 (time 1784341647.49 total 79.48 last 163.37) - open/close/unlink 14948 (time 1784341657.49 total 89.48 last 173.86) - open/close/unlink 16704 (time 1784341667.50 total 99.49 last 175.52) - open/close/unlink 18458 (time 1784341677.50 total 109.49 last 175.31) - open/close/unlink 20000 (time 1784341686.33 total 118.32 last 174.63) - open/close/unlink 21558 (time 1784341696.33 total 128.33 last 155.73) - open/close/unlink 23324 (time 1784341706.34 total 138.33 last 176.55) - open/close/unlink 25072 (time 1784341716.34 total 148.33 last 174.80) - open/close/unlink 26814 (time 1784341726.34 total 158.33 last 174.15) - open/close/unlink 28545 (time 1784341736.35 total 168.34 last 173.03) - open/close/unlink 30000 (time 1784341745.35 total 177.34 last 161.61) - open/close/unlink 31627 (time 1784341755.35 total 187.34 last 162.68) - open/close/unlink 33229 (time 1784341765.36 total 197.35 last 160.04) - open/close/unlink 34771 (time 1784341775.37 total 207.36 last 154.12) - open/close/unlink 36475 (time 1784341785.38 total 217.37 last 170.23) - open/close/unlink 38235 (time 1784341795.38 total 227.37 last 175.90) - open/close/unlink 39995 (time 1784341805.38 total 237.38 last 175.96) - open/close/unlink 41679 (time 1784341815.38 total 247.38 last 168.37) - open/close/unlink 43439 (time 1784341825.39 total 257.38 last 175.97) - open/close/unlink 45182 (time 1784341835.39 total 267.38 last 174.23) - open/close/unlink 46870 (time 1784341845.39 total 277.38 last 168.76) - open/close/unlink 48542 (time 1784341855.40 total 287.39 last 167.11) - open/close/unlink 50000 (time 1784341864.38 total 296.37 last 162.30) - open/close/unlink 51620 (time 1784341874.38 total 306.38 last 161.95) - open/close/unlink 53262 (time 1784341884.39 total 316.38 last 164.18) - open/close/unlink 54937 (time 1784341894.39 total 326.38 last 167.47) - open/close/unlink 56689 (time 1784341904.39 total 336.39 last 175.10) - open/close/unlink 58200 (time 1784341914.40 total 346.39 last 151.02) - open/close/unlink 59868 (time 1784341924.41 total 356.40 last 166.66) - open/close/unlink 61529 (time 1784341934.41 total 366.40 last 166.08) - open/close/unlink 63203 (time 1784341944.41 total 376.41 last 167.31) total: 64768 open/close/unlink in 385.68 seconds: 167.93 ops/second - open/close/unlink 1753 (time 1784341578.09 total 10.00 last 175.22) - open/close/unlink 3420 (time 1784341588.09 total 20.00 last 166.70) - open/close/unlink 5093 (time 1784341598.09 total 30.01 last 167.29) - open/close/unlink 6752 (time 1784341608.09 total 40.01 last 165.84) - open/close/unlink 8407 (time 1784341618.10 total 50.01 last 165.42) - open/close/unlink 10000 (time 1784341627.91 total 59.83 last 162.32) - open/close/unlink 11596 (time 1784341637.92 total 69.83 last 159.49) - open/close/unlink 13225 (time 1784341647.92 total 79.84 last 162.84) - open/close/unlink 14946 (time 1784341657.94 total 89.85 last 171.89) - open/close/unlink 16689 (time 1784341667.94 total 99.85 last 174.28) - open/close/unlink 18417 (time 1784341677.94 total 109.86 last 172.70) - open/close/unlink 20000 (time 1784341686.78 total 118.70 last 179.08) - open/close/unlink 21561 (time 1784341696.79 total 128.70 last 156.05) - open/close/unlink 23323 (time 1784341706.79 total 138.70 last 176.13) - open/close/unlink 25071 (time 1784341716.79 total 148.71 last 174.76) - open/close/unlink 26801 (time 1784341726.79 total 158.71 last 172.98) - open/close/unlink 28537 (time 1784341736.80 total 168.71 last 173.53) - open/close/unlink 30000 (time 1784341745.78 total 177.69 last 162.89) - open/close/unlink 31605 (time 1784341755.78 total 187.70 last 160.46) - open/close/unlink 33191 (time 1784341765.79 total 197.70 last 158.55) - open/close/unlink 34710 (time 1784341775.79 total 207.70 last 151.83) - open/close/unlink 36430 (time 1784341785.85 total 217.77 last 170.92) - open/close/unlink 38201 (time 1784341795.86 total 227.77 last 177.04) - open/close/unlink 39974 (time 1784341805.86 total 237.78 last 177.22) - open/close/unlink 41692 (time 1784341815.86 total 247.78 last 171.74) - open/close/unlink 43462 (time 1784341825.87 total 257.78 last 176.95) - open/close/unlink 45194 (time 1784341835.87 total 267.79 last 173.14) - open/close/unlink 46879 (time 1784341845.87 total 277.79 last 168.46) - open/close/unlink 48542 (time 1784341855.87 total 287.79 last 166.30) - open/close/unlink 50000 (time 1784341864.79 total 296.70 last 163.54) - open/close/unlink 51627 (time 1784341874.79 total 306.70 last 162.70) - open/close/unlink 53271 (time 1784341884.79 total 316.71 last 164.34) - open/close/unlink 54957 (time 1784341894.79 total 326.71 last 168.59) - open/close/unlink 56677 (time 1784341904.79 total 336.71 last 171.99) - open/close/unlink 58213 (time 1784341914.79 total 346.71 last 153.58) - open/close/unlink 59837 (time 1784341924.80 total 356.71 last 162.35) - open/close/unlink 61523 (time 1784341934.80 total 366.71 last 168.55) - open/close/unlink 63184 (time 1784341944.80 total 376.72 last 166.08) - open/close/unlink 64749 (time 1784341954.82 total 386.73 last 156.28) total: 64768 open/close/unlink in 387.29 seconds: 167.23 ops/second - open/close/unlink 1773 (time 1784341578.21 total 10.00 last 177.29) - open/close/unlink 3449 (time 1784341588.21 total 20.00 last 167.57) - open/close/unlink 5118 (time 1784341598.21 total 30.00 last 166.90) - open/close/unlink 6785 (time 1784341608.21 total 40.00 last 166.66) - open/close/unlink 8457 (time 1784341618.21 total 50.01 last 167.12) - open/close/unlink 10000 (time 1784341627.76 total 59.55 last 161.67) - open/close/unlink 11579 (time 1784341637.77 total 69.56 last 157.80) - open/close/unlink 13223 (time 1784341647.77 total 79.56 last 164.35) - open/close/unlink 14951 (time 1784341657.78 total 89.57 last 172.67) - open/close/unlink 16691 (time 1784341667.78 total 99.58 last 173.87) - open/close/unlink 18425 (time 1784341677.79 total 109.58 last 173.36) - open/close/unlink 20000 (time 1784341686.68 total 118.48 last 177.00) - open/close/unlink 21547 (time 1784341696.69 total 128.48 last 154.65) - open/close/unlink 23309 (time 1784341706.69 total 138.48 last 176.16) - open/close/unlink 25034 (time 1784341716.69 total 148.49 last 172.42) - open/close/unlink 26765 (time 1784341726.70 total 158.49 last 173.07) - open/close/unlink 28491 (time 1784341736.70 total 168.50 last 172.51) - open/close/unlink 30000 (time 1784341745.91 total 177.70 last 163.92) - open/close/unlink 31626 (time 1784341755.91 total 187.70 last 162.57) - open/close/unlink 33209 (time 1784341765.91 total 197.71 last 158.27) - open/close/unlink 34731 (time 1784341775.91 total 207.71 last 152.14) - open/close/unlink 36412 (time 1784341785.93 total 217.72 last 167.93) - open/close/unlink 38179 (time 1784341795.93 total 227.72 last 176.65) - open/close/unlink 39951 (time 1784341805.93 total 237.72 last 177.19) - open/close/unlink 41661 (time 1784341815.94 total 247.73 last 170.87) - open/close/unlink 43437 (time 1784341825.94 total 257.74 last 177.50) - open/close/unlink 45180 (time 1784341835.94 total 267.74 last 174.27) - open/close/unlink 46860 (time 1784341845.95 total 277.74 last 167.96) - open/close/unlink 48514 (time 1784341855.95 total 287.75 last 165.31) - open/close/unlink 50000 (time 1784341865.02 total 296.81 last 163.89) - open/close/unlink 51623 (time 1784341875.02 total 306.82 last 162.25) - open/close/unlink 53267 (time 1784341885.02 total 316.82 last 164.35) - open/close/unlink 54973 (time 1784341895.03 total 326.82 last 170.56) - open/close/unlink 56706 (time 1784341905.03 total 336.83 last 173.21) - open/close/unlink 58223 (time 1784341915.03 total 346.83 last 151.68) - open/close/unlink 59835 (time 1784341925.03 total 356.83 last 161.18) - open/close/unlink 61531 (time 1784341935.04 total 366.83 last 169.49) - open/close/unlink 63172 (time 1784341945.04 total 376.84 last 164.08) - open/close/unlink 64713 (time 1784341955.07 total 386.87 last 153.59) total: 64768 open/close/unlink in 388.29 seconds: 166.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg204-server: cannot open 'lustre-ost2': no such pool pdsh@oleg204-client: oleg204-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2