== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:42:20 (1786934540) start mds service on oleg452-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg452-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg452-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 2011 (time 1786934584.32 total 10.00 last 201.03) - open/close/unlink 4175 (time 1786934594.32 total 20.00 last 216.38) - open/close/unlink 6406 (time 1786934604.33 total 30.01 last 223.04) - open/close/unlink 8510 (time 1786934614.34 total 40.02 last 210.22) - open/close/unlink 10000 (time 1786934620.97 total 46.65 last 224.70) - open/close/unlink 12236 (time 1786934630.97 total 56.65 last 223.51) - open/close/unlink 14385 (time 1786934640.97 total 66.65 last 214.85) - open/close/unlink 16466 (time 1786934650.98 total 76.66 last 208.04) - open/close/unlink 18483 (time 1786934660.98 total 86.66 last 201.64) - open/close/unlink 20000 (time 1786934668.38 total 94.06 last 204.85) - open/close/unlink 22025 (time 1786934678.39 total 104.07 last 202.42) - open/close/unlink 24110 (time 1786934688.39 total 114.07 last 208.48) - open/close/unlink 26205 (time 1786934698.39 total 124.07 last 209.47) - open/close/unlink 28207 (time 1786934708.39 total 134.07 last 200.14) - open/close/unlink 30000 (time 1786934717.16 total 142.84 last 204.51) - open/close/unlink 32058 (time 1786934727.16 total 152.84 last 205.75) - open/close/unlink 34097 (time 1786934737.17 total 162.85 last 203.80) - open/close/unlink 36159 (time 1786934747.17 total 172.85 last 206.15) - open/close/unlink 38216 (time 1786934757.17 total 182.85 last 205.64) - open/close/unlink 40000 (time 1786934765.95 total 191.63 last 203.17) - open/close/unlink 42071 (time 1786934775.95 total 201.63 last 207.09) - open/close/unlink 44112 (time 1786934785.96 total 211.64 last 204.08) - open/close/unlink 46253 (time 1786934795.96 total 221.64 last 214.03) - open/close/unlink 48469 (time 1786934805.96 total 231.64 last 221.58) - open/close/unlink 50000 (time 1786934813.05 total 238.73 last 215.93) - open/close/unlink 52084 (time 1786934823.05 total 248.73 last 208.40) - open/close/unlink 54170 (time 1786934833.05 total 258.73 last 208.60) - open/close/unlink 56257 (time 1786934843.05 total 268.73 last 208.64) - open/close/unlink 58375 (time 1786934853.06 total 278.74 last 211.74) - open/close/unlink 60000 (time 1786934861.23 total 286.91 last 198.69) - open/close/unlink 62143 (time 1786934871.24 total 296.92 last 214.26) - open/close/unlink 64233 (time 1786934881.24 total 306.92 last 208.93) total: 64768 open/close/unlink in 309.64 seconds: 209.17 ops/second - open/close/unlink 1950 (time 1786934584.30 total 10.00 last 194.99) - open/close/unlink 4111 (time 1786934594.30 total 20.00 last 216.07) - open/close/unlink 6322 (time 1786934604.31 total 30.01 last 220.99) - open/close/unlink 8410 (time 1786934614.31 total 40.01 last 208.75) - open/close/unlink 10000 (time 1786934621.48 total 47.17 last 221.94) - open/close/unlink 12220 (time 1786934631.48 total 57.18 last 221.96) - open/close/unlink 14372 (time 1786934641.48 total 67.18 last 215.20) - open/close/unlink 16456 (time 1786934651.48 total 77.18 last 208.38) - open/close/unlink 18453 (time 1786934661.48 total 87.18 last 199.62) - open/close/unlink 20000 (time 1786934668.97 total 94.67 last 206.62) - open/close/unlink 22051 (time 1786934678.97 total 104.67 last 205.08) - open/close/unlink 24137 (time 1786934688.97 total 114.67 last 208.56) - open/close/unlink 26195 (time 1786934698.98 total 124.67 last 205.72) - open/close/unlink 28166 (time 1786934708.98 total 134.68 last 197.05) - open/close/unlink 30000 (time 1786934717.81 total 143.51 last 207.60) - open/close/unlink 32043 (time 1786934727.81 total 153.51 last 204.28) - open/close/unlink 34115 (time 1786934737.82 total 163.51 last 207.19) - open/close/unlink 36171 (time 1786934747.82 total 173.52 last 205.52) - open/close/unlink 38206 (time 1786934757.82 total 183.52 last 203.47) - open/close/unlink 40000 (time 1786934766.51 total 192.21 last 206.36) - open/close/unlink 42010 (time 1786934776.52 total 202.21 last 200.97) - open/close/unlink 44056 (time 1786934786.52 total 212.22 last 204.55) - open/close/unlink 46239 (time 1786934796.52 total 222.22 last 218.25) - open/close/unlink 48492 (time 1786934806.52 total 232.22 last 225.27) - open/close/unlink 50000 (time 1786934813.73 total 239.42 last 209.31) - open/close/unlink 52100 (time 1786934823.73 total 249.43 last 209.97) - open/close/unlink 54174 (time 1786934833.73 total 259.43 last 207.36) - open/close/unlink 56272 (time 1786934843.73 total 269.43 last 209.78) - open/close/unlink 58419 (time 1786934853.73 total 279.43 last 214.62) - open/close/unlink 60000 (time 1786934861.57 total 287.26 last 201.85) - open/close/unlink 62166 (time 1786934871.57 total 297.27 last 216.55) - open/close/unlink 64279 (time 1786934881.57 total 307.27 last 211.26) total: 64768 open/close/unlink in 310.19 seconds: 208.80 ops/second - open/close/unlink 1985 (time 1786934584.08 total 10.00 last 198.49) - open/close/unlink 4102 (time 1786934594.08 total 20.00 last 211.66) - open/close/unlink 6313 (time 1786934604.08 total 30.01 last 221.04) - open/close/unlink 8423 (time 1786934614.08 total 40.01 last 210.96) - open/close/unlink 10000 (time 1786934621.44 total 47.37 last 214.22) - open/close/unlink 12217 (time 1786934631.45 total 57.37 last 221.66) - open/close/unlink 14393 (time 1786934641.45 total 67.37 last 217.53) - open/close/unlink 16474 (time 1786934651.45 total 77.37 last 208.07) - open/close/unlink 18491 (time 1786934661.45 total 87.38 last 201.67) - open/close/unlink 20000 (time 1786934668.71 total 94.63 last 207.97) - open/close/unlink 22041 (time 1786934678.71 total 104.63 last 204.06) - open/close/unlink 24094 (time 1786934688.71 total 114.63 last 205.28) - open/close/unlink 26158 (time 1786934698.71 total 124.64 last 206.35) - open/close/unlink 28162 (time 1786934708.71 total 134.64 last 200.37) - open/close/unlink 30000 (time 1786934717.76 total 143.69 last 203.14) - open/close/unlink 32068 (time 1786934727.76 total 153.69 last 206.79) - open/close/unlink 34102 (time 1786934737.77 total 163.69 last 203.34) - open/close/unlink 36168 (time 1786934747.77 total 173.69 last 206.55) - open/close/unlink 38175 (time 1786934757.77 total 183.70 last 200.63) - open/close/unlink 40000 (time 1786934766.57 total 192.50 last 207.41) - open/close/unlink 42023 (time 1786934776.57 total 202.50 last 202.23) - open/close/unlink 44052 (time 1786934786.58 total 212.50 last 202.88) - open/close/unlink 46281 (time 1786934796.58 total 222.50 last 222.85) - open/close/unlink 48461 (time 1786934806.58 total 232.51 last 217.94) - open/close/unlink 50000 (time 1786934813.83 total 239.76 last 212.27) - open/close/unlink 52124 (time 1786934823.83 total 249.76 last 212.35) - open/close/unlink 54235 (time 1786934833.83 total 259.76 last 211.09) - open/close/unlink 56308 (time 1786934843.84 total 269.76 last 207.23) - open/close/unlink 58441 (time 1786934853.84 total 279.76 last 213.29) - open/close/unlink 60000 (time 1786934861.87 total 287.80 last 194.07) - open/close/unlink 62163 (time 1786934871.87 total 297.80 last 216.28) - open/close/unlink 64246 (time 1786934881.87 total 307.80 last 208.26) total: 64768 open/close/unlink in 312.52 seconds: 207.24 ops/second - open/close/unlink 1937 (time 1786934584.29 total 10.01 last 193.57) - open/close/unlink 4052 (time 1786934594.29 total 20.01 last 211.45) - open/close/unlink 6283 (time 1786934604.29 total 30.01 last 223.02) - open/close/unlink 8373 (time 1786934614.29 total 40.01 last 208.99) - open/close/unlink 10000 (time 1786934621.69 total 47.41 last 219.93) - open/close/unlink 12253 (time 1786934631.70 total 57.42 last 225.21) - open/close/unlink 14361 (time 1786934641.70 total 67.42 last 210.77) - open/close/unlink 16424 (time 1786934651.70 total 77.42 last 206.22) - open/close/unlink 18398 (time 1786934661.70 total 87.42 last 197.35) - open/close/unlink 20000 (time 1786934669.38 total 95.10 last 208.69) - open/close/unlink 22068 (time 1786934679.39 total 105.10 last 206.69) - open/close/unlink 24129 (time 1786934689.39 total 115.11 last 206.00) - open/close/unlink 26163 (time 1786934699.40 total 125.11 last 203.30) - open/close/unlink 28146 (time 1786934709.41 total 135.12 last 198.12) - open/close/unlink 30000 (time 1786934718.59 total 144.31 last 201.84) - open/close/unlink 32097 (time 1786934728.59 total 154.31 last 209.67) - open/close/unlink 34150 (time 1786934738.59 total 164.31 last 205.28) - open/close/unlink 36159 (time 1786934748.60 total 174.31 last 200.85) - open/close/unlink 38212 (time 1786934758.60 total 184.32 last 205.21) - open/close/unlink 40000 (time 1786934767.12 total 192.84 last 209.76) - open/close/unlink 42065 (time 1786934777.13 total 202.85 last 206.39) - open/close/unlink 44106 (time 1786934787.13 total 212.85 last 204.08) - open/close/unlink 46312 (time 1786934797.13 total 222.85 last 220.53) - open/close/unlink 48552 (time 1786934807.13 total 232.85 last 223.97) - open/close/unlink 50000 (time 1786934813.97 total 239.69 last 211.82) - open/close/unlink 52097 (time 1786934823.97 total 249.69 last 209.67) - open/close/unlink 54175 (time 1786934833.97 total 259.69 last 207.75) - open/close/unlink 56235 (time 1786934843.98 total 269.70 last 205.95) - open/close/unlink 58391 (time 1786934853.98 total 279.70 last 215.50) - open/close/unlink 60000 (time 1786934862.17 total 287.89 last 196.52) - open/close/unlink 62178 (time 1786934872.17 total 297.89 last 217.73) - open/close/unlink 64269 (time 1786934882.17 total 307.89 last 209.08) total: 64768 open/close/unlink in 312.97 seconds: 206.95 ops/second - open/close/unlink 1981 (time 1786934584.31 total 10.00 last 198.02) - open/close/unlink 4097 (time 1786934594.32 total 20.01 last 211.52) - open/close/unlink 6288 (time 1786934604.32 total 30.01 last 219.08) - open/close/unlink 8398 (time 1786934614.33 total 40.02 last 210.86) - open/close/unlink 10000 (time 1786934621.50 total 47.19 last 223.36) - open/close/unlink 12205 (time 1786934631.50 total 57.19 last 220.47) - open/close/unlink 14329 (time 1786934641.50 total 67.19 last 212.38) - open/close/unlink 16408 (time 1786934651.50 total 77.19 last 207.83) - open/close/unlink 18398 (time 1786934661.51 total 87.20 last 198.95) - open/close/unlink 20000 (time 1786934669.24 total 94.93 last 207.04) - open/close/unlink 22020 (time 1786934679.25 total 104.94 last 201.86) - open/close/unlink 24090 (time 1786934689.26 total 114.95 last 206.89) - open/close/unlink 26179 (time 1786934699.26 total 124.95 last 208.85) - open/close/unlink 28219 (time 1786934709.26 total 134.95 last 203.89) - open/close/unlink 30000 (time 1786934717.86 total 143.55 last 207.22) - open/close/unlink 32033 (time 1786934727.86 total 153.55 last 203.20) - open/close/unlink 34098 (time 1786934737.86 total 163.55 last 206.48) - open/close/unlink 36176 (time 1786934747.87 total 173.56 last 207.77) - open/close/unlink 38164 (time 1786934757.87 total 183.56 last 198.76) - open/close/unlink 40000 (time 1786934766.76 total 192.45 last 206.42) - open/close/unlink 42049 (time 1786934776.76 total 202.45 last 204.86) - open/close/unlink 44076 (time 1786934786.77 total 212.46 last 202.68) - open/close/unlink 46254 (time 1786934796.77 total 222.46 last 217.77) - open/close/unlink 48430 (time 1786934806.77 total 232.46 last 217.53) - open/close/unlink 50000 (time 1786934814.26 total 239.95 last 209.73) - open/close/unlink 52082 (time 1786934824.26 total 249.95 last 208.17) - open/close/unlink 54170 (time 1786934834.26 total 259.95 last 208.75) - open/close/unlink 56227 (time 1786934844.26 total 269.95 last 205.69) - open/close/unlink 58347 (time 1786934854.26 total 279.95 last 211.93) - open/close/unlink 60000 (time 1786934862.65 total 288.34 last 197.05) - open/close/unlink 62166 (time 1786934872.66 total 298.34 last 216.53) - open/close/unlink 64246 (time 1786934882.66 total 308.35 last 207.97) total: 64768 open/close/unlink in 314.08 seconds: 206.22 ops/second Waiting for MDT destroys to complete oleg452-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg452-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg452-server unloading modules via unload_modules_local on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing unload_modules_local oleg452-server: modules unloaded. pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2