== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:47:14 (1734259634) Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg116-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg116-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1737 (time 1734259696.50 total 10.00 last 173.63) - open/close 3384 (time 1734259706.51 total 20.01 last 164.66) - open/close 4938 (time 1734259716.51 total 30.01 last 155.35) - open/close 6485 (time 1734259726.51 total 40.01 last 154.70) - open/close 8115 (time 1734259736.52 total 50.02 last 162.90) - open/close 9921 (time 1734259746.52 total 60.02 last 180.53) - open/close 11583 (time 1734259756.52 total 70.02 last 166.18) - open/close 13239 (time 1734259766.52 total 80.02 last 165.58) - open/close 15071 (time 1734259776.53 total 90.02 last 183.16) - open/close 16988 (time 1734259786.53 total 100.03 last 191.64) - open/close 19042 (time 1734259796.53 total 110.03 last 205.33) - open/close 20000 (time 1734259801.99 total 115.49 last 175.37) - open/close 21825 (time 1734259812.00 total 125.50 last 182.48) - open/close 23720 (time 1734259822.00 total 135.50 last 189.41) - open/close 25417 (time 1734259832.00 total 145.50 last 169.64) - open/close 27055 (time 1734259842.01 total 155.51 last 163.78) - open/close 28691 (time 1734259852.01 total 165.51 last 163.54) - open/close 30000 (time 1734259859.20 total 172.70 last 182.02) - open/close 31580 (time 1734259869.20 total 182.70 last 157.94) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 188.77 seconds: 172.49 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3123 (time 1734259887.02 total 10.00 last 312.26) - /unlink 6366 (time 1734259897.02 total 20.00 last 324.22) - /unlink 9794 (time 1734259907.02 total 30.00 last 342.80) - /unlink 13428 (time 1734259917.02 total 40.00 last 363.36) - /unlink 17200 (time 1734259927.02 total 50.01 last 377.13) - /unlink 20000 (time 1734259934.44 total 57.43 last 377.38) - /unlink 23740 (time 1734259944.44 total 67.43 last 373.91) - /unlink 27546 (time 1734259954.45 total 77.43 last 380.54) - /unlink 30000 (time 1734259960.55 total 83.54 last 401.99) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 90.04 seconds: 361.62 ops/second Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 - open/close 2090 (time 1734259988.57 total 10.00 last 209.00) - open/close 4239 (time 1734259998.57 total 20.00 last 214.88) - open/close 6473 (time 1734260008.57 total 30.00 last 223.34) - open/close 8593 (time 1734260018.57 total 40.00 last 211.99) - open/close 10000 (time 1734260025.32 total 46.76 last 208.30) - open/close 12048 (time 1734260035.33 total 56.76 last 204.76) - open/close 14107 (time 1734260045.33 total 66.76 last 205.82) - open/close 16156 (time 1734260055.33 total 76.77 last 204.86) - open/close 18070 (time 1734260065.34 total 86.77 last 191.32) - open/close 19781 (time 1734260075.34 total 96.77 last 171.07) - open/close 21541 (time 1734260085.34 total 106.77 last 175.98) - open/close 23574 (time 1734260095.34 total 116.78 last 203.23) - open/close 25439 (time 1734260105.34 total 126.78 last 186.47) - open/close 27425 (time 1734260115.35 total 136.78 last 198.56) - open/close 29361 (time 1734260125.35 total 146.78 last 193.52) - open/close 30000 (time 1734260128.20 total 149.63 last 224.51) - open/close 32181 (time 1734260138.20 total 159.63 last 218.09) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 161.37 seconds: 201.77 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3943 (time 1734260151.08 total 10.00 last 394.27) - /unlink 8570 (time 1734260161.08 total 20.00 last 462.63) - /unlink 10000 (time 1734260164.24 total 23.16 last 452.17) - /unlink 14362 (time 1734260174.24 total 33.17 last 436.15) - /unlink 19436 (time 1734260184.25 total 43.17 last 507.30) - /unlink 24875 (time 1734260194.25 total 53.17 last 543.80) - /unlink 29842 (time 1734260204.25 total 63.17 last 496.67) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 67.93 seconds: 479.34 ops/second Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 - open/close 3175 (time 1734260228.41 total 10.00 last 317.43) - open/close 6216 (time 1734260238.42 total 20.01 last 303.96) - open/close 8874 (time 1734260248.42 total 30.01 last 265.73) - open/close 10000 (time 1734260252.65 total 34.24 last 266.12) - open/close 12772 (time 1734260262.65 total 44.24 last 277.19) - open/close 15683 (time 1734260272.65 total 54.24 last 291.08) - open/close 18877 (time 1734260282.65 total 64.24 last 319.33) - open/close 20000 (time 1734260286.14 total 67.73 last 322.57) - open/close 22999 (time 1734260296.14 total 77.73 last 299.81) - open/close 25577 (time 1734260306.14 total 87.73 last 257.74) - open/close 27967 (time 1734260316.14 total 97.73 last 238.97) - open/close 30000 (time 1734260325.03 total 106.62 last 228.87) - open/close 32555 (time 1734260335.03 total 116.62 last 255.43) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 116.65 seconds: 279.14 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4143 (time 1734260346.28 total 10.00 last 414.24) - /unlink 9193 (time 1734260356.28 total 20.00 last 504.98) - /unlink 15370 (time 1734260366.28 total 30.00 last 617.65) - /unlink 20000 (time 1734260374.86 total 38.58 last 539.97) - /unlink 25523 (time 1734260384.86 total 48.58 last 552.28) - /unlink 30000 (time 1734260392.61 total 56.33 last 577.76) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 61.24 seconds: 531.72 ops/second Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 - open/close 2657 (time 1734260434.94 total 10.00 last 265.68) - open/close 4971 (time 1734260444.94 total 20.00 last 231.36) - open/close 7352 (time 1734260454.94 total 30.00 last 238.07) - open/close 9787 (time 1734260464.94 total 40.00 last 243.49) - open/close 11985 (time 1734260474.94 total 50.01 last 219.75) - open/close 14295 (time 1734260484.95 total 60.01 last 230.92) - open/close 16510 (time 1734260494.95 total 70.01 last 221.42) - open/close 18868 (time 1734260504.95 total 80.01 last 235.77) - open/close 20000 (time 1734260509.24 total 84.30 last 263.92) - open/close 22813 (time 1734260519.24 total 94.31 last 281.20) - open/close 25880 (time 1734260529.25 total 104.31 last 306.66) - open/close 28851 (time 1734260539.25 total 114.31 last 297.07) - open/close 30000 (time 1734260543.40 total 118.46 last 276.73) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 127.42 seconds: 255.54 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5024 (time 1734260563.42 total 10.00 last 502.39) - /unlink 10000 (time 1734260571.50 total 18.08 last 615.61) - /unlink 16220 (time 1734260581.50 total 28.08 last 621.98) - /unlink 20000 (time 1734260587.32 total 33.90 last 649.50) - /unlink 25389 (time 1734260597.32 total 43.91 last 538.81) - /unlink 30000 (time 1734260604.78 total 51.36 last 618.24) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 55.43 seconds: 587.44 ops/second Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 - open/close 4027 (time 1734260644.69 total 10.00 last 402.65) - open/close 6781 (time 1734260654.69 total 20.00 last 275.32) - open/close 9567 (time 1734260664.69 total 30.01 last 278.51) - open/close 12544 (time 1734260674.69 total 40.01 last 297.66) - open/close 15559 (time 1734260684.69 total 50.01 last 301.50) - open/close 18773 (time 1734260694.70 total 60.01 last 321.39) - open/close 20000 (time 1734260698.65 total 63.96 last 310.28) - open/close 23104 (time 1734260708.65 total 73.97 last 310.35) - open/close 26274 (time 1734260718.65 total 83.97 last 316.95) - open/close 29661 (time 1734260728.66 total 93.97 last 338.61) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 102.13 seconds: 318.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32688 47304 41% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32943 32943 0 100% /mnt/lustre - /unlink 6710 (time 1734260747.83 total 10.00 last 670.88) - /unlink 10000 (time 1734260752.30 total 14.47 last 736.29) - /unlink 18554 (time 1734260762.30 total 24.47 last 855.39) - /unlink 27764 (time 1734260772.30 total 34.47 last 920.93) - /unlink 30000 (time 1734260774.77 total 36.94 last 906.04) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 40.11 seconds: 811.84 ops/second Waiting for MDT destroys to complete oleg116-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg116-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2