== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:13:09 (1734430389) Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f) oleg205-server: oleg205-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: detected 4 online CPUs by sysfs oleg205-server: MODOPTS_LIBCFS= oleg205-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg205-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg205-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg205-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1490 (time 1734430452.68 total 10.00 last 148.98) - open/close 3045 (time 1734430462.69 total 20.00 last 155.46) - open/close 4542 (time 1734430472.69 total 30.01 last 149.66) - open/close 5908 (time 1734430482.69 total 40.01 last 136.57) - open/close 7612 (time 1734430492.70 total 50.01 last 170.33) - open/close 9591 (time 1734430502.70 total 60.02 last 197.81) - open/close 10000 (time 1734430505.19 total 62.51 last 164.11) - open/close 11766 (time 1734430515.20 total 72.51 last 176.54) - open/close 13505 (time 1734430525.20 total 82.52 last 173.83) - open/close 14996 (time 1734430535.20 total 92.52 last 149.05) - open/close 16407 (time 1734430545.21 total 102.52 last 141.08) - open/close 17861 (time 1734430555.21 total 112.52 last 145.36) - open/close 19265 (time 1734430565.21 total 122.52 last 140.40) - open/close 20000 (time 1734430570.40 total 127.71 last 141.68) - open/close 21524 (time 1734430580.40 total 137.71 last 152.37) - open/close 22882 (time 1734430590.40 total 147.72 last 135.79) - open/close 24498 (time 1734430600.40 total 157.72 last 161.58) - open/close 26208 (time 1734430610.41 total 167.73 last 170.83) - open/close 27945 (time 1734430620.42 total 177.73 last 173.59) - open/close 29574 (time 1734430630.42 total 187.74 last 162.82) - open/close 30000 (time 1734430633.70 total 191.02 last 129.94) - open/close 31499 (time 1734430643.70 total 201.02 last 149.87) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 208.47 seconds: 156.19 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 2971 (time 1734430662.74 total 10.00 last 297.08) - /unlink 5641 (time 1734430672.75 total 20.00 last 266.95) - /unlink 8152 (time 1734430682.75 total 30.01 last 251.02) - /unlink 10000 (time 1734430689.52 total 36.78 last 272.91) - /unlink 12493 (time 1734430699.52 total 46.78 last 249.28) - /unlink 15150 (time 1734430709.52 total 56.78 last 265.61) - /unlink 17879 (time 1734430719.53 total 66.78 last 272.86) - /unlink 20000 (time 1734430726.65 total 73.91 last 297.56) - /unlink 22698 (time 1734430736.66 total 83.91 last 269.76) - /unlink 25418 (time 1734430746.66 total 93.92 last 271.85) - /unlink 28200 (time 1734430756.67 total 103.92 last 278.09) - /unlink 30000 (time 1734430763.04 total 110.29 last 282.51) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 118.63 seconds: 274.47 ops/second Waiting for MDT destroys to complete oleg205-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 oleg205-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 - open/close 1696 (time 1734430793.74 total 10.00 last 169.52) - open/close 3254 (time 1734430803.74 total 20.01 last 155.78) - open/close 5124 (time 1734430813.74 total 30.01 last 186.99) - open/close 7063 (time 1734430823.74 total 40.01 last 193.83) - open/close 9125 (time 1734430833.74 total 50.01 last 206.19) - open/close 10000 (time 1734430838.15 total 54.42 last 198.56) - open/close 12153 (time 1734430848.16 total 64.42 last 215.17) - open/close 14239 (time 1734430858.16 total 74.43 last 208.54) - open/close 16306 (time 1734430868.16 total 84.43 last 206.68) - open/close 18136 (time 1734430878.16 total 94.43 last 183.00) - open/close 20000 (time 1734430885.12 total 101.39 last 267.70) - open/close 22354 (time 1734430895.12 total 111.39 last 235.40) - open/close 24883 (time 1734430905.13 total 121.39 last 252.86) - open/close 27405 (time 1734430915.13 total 131.40 last 252.11) - open/close 29297 (time 1734430925.13 total 141.40 last 189.10) - open/close 30000 (time 1734430928.30 total 144.56 last 222.40) - open/close 31959 (time 1734430938.30 total 154.56 last 195.89) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 157.82 seconds: 206.31 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3611 (time 1734430952.81 total 10.00 last 361.03) - /unlink 6621 (time 1734430962.81 total 20.00 last 301.00) - /unlink 9611 (time 1734430972.82 total 30.00 last 298.96) - /unlink 12801 (time 1734430982.82 total 40.01 last 318.95) - /unlink 15823 (time 1734430992.82 total 50.01 last 302.17) - /unlink 19034 (time 1734431002.82 total 60.01 last 321.09) - /unlink 20000 (time 1734431005.36 total 62.55 last 380.20) - /unlink 23178 (time 1734431015.36 total 72.55 last 317.77) - /unlink 26175 (time 1734431025.36 total 82.55 last 299.62) - /unlink 29384 (time 1734431035.36 total 92.55 last 320.89) - /unlink 30000 (time 1734431037.57 total 94.75 last 279.53) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 103.41 seconds: 314.87 ops/second Waiting for MDT destroys to complete oleg205-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg205-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 - open/close 1811 (time 1734431068.09 total 10.00 last 181.03) - open/close 3309 (time 1734431078.09 total 20.01 last 149.74) - open/close 4626 (time 1734431088.10 total 30.01 last 131.63) - open/close 6174 (time 1734431098.11 total 40.02 last 154.70) - open/close 7588 (time 1734431108.11 total 50.03 last 141.28) - open/close 8952 (time 1734431118.12 total 60.03 last 136.38) - open/close 10000 (time 1734431124.43 total 66.34 last 166.04) - open/close 11466 (time 1734431134.43 total 76.35 last 146.52) - open/close 12743 (time 1734431144.44 total 86.35 last 127.64) - open/close 14138 (time 1734431154.44 total 96.35 last 139.48) - open/close 15649 (time 1734431164.44 total 106.36 last 151.05) - open/close 17369 (time 1734431174.44 total 116.36 last 171.97) - open/close 19134 (time 1734431184.45 total 126.36 last 176.39) - open/close 20000 (time 1734431189.42 total 131.33 last 174.41) - open/close 21549 (time 1734431199.42 total 141.33 last 154.87) - open/close 23070 (time 1734431209.42 total 151.33 last 152.05) - open/close 25552 (time 1734431219.42 total 161.34 last 248.16) - open/close 27651 (time 1734431229.43 total 171.34 last 209.82) - open/close 29322 (time 1734431239.43 total 181.34 last 167.09) - open/close 30000 (time 1734431243.67 total 185.58 last 159.81) - open/close 31617 (time 1734431253.67 total 195.59 last 161.62) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 201.87 seconds: 161.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 2583 (time 1734431271.62 total 10.00 last 258.27) - /unlink 5209 (time 1734431281.62 total 20.00 last 262.53) - /unlink 8202 (time 1734431291.63 total 30.01 last 299.22) - /unlink 10000 (time 1734431298.29 total 36.67 last 269.82) - /unlink 12979 (time 1734431308.29 total 46.67 last 297.90) - /unlink 15672 (time 1734431318.29 total 56.67 last 269.29) - /unlink 18202 (time 1734431328.29 total 66.67 last 252.98) - /unlink 20000 (time 1734431335.55 total 73.93 last 247.64) - /unlink 22596 (time 1734431345.55 total 83.93 last 259.57) - /unlink 25413 (time 1734431355.56 total 93.94 last 281.55) - /unlink 27932 (time 1734431365.56 total 103.94 last 251.88) - /unlink 30000 (time 1734431373.71 total 112.09 last 253.69) - /unlink 32518 (time 1734431383.71 total 122.09 last 251.75) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 122.22 seconds: 266.40 ops/second Waiting for MDT destroys to complete oleg205-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 oleg205-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 - open/close 1371 (time 1734431404.31 total 10.00 last 137.06) - open/close 2844 (time 1734431414.31 total 20.01 last 147.21) - open/close 4280 (time 1734431424.32 total 30.01 last 143.55) - open/close 5856 (time 1734431434.32 total 40.02 last 157.57) - open/close 7381 (time 1734431444.32 total 50.02 last 152.45) - open/close 8726 (time 1734431454.33 total 60.03 last 134.40) - open/close 10000 (time 1734431462.56 total 68.26 last 154.75) - open/close 11507 (time 1734431472.56 total 78.26 last 150.66) - open/close 12814 (time 1734431482.56 total 88.26 last 130.69) - open/close 14257 (time 1734431492.57 total 98.27 last 144.19) - open/close 15715 (time 1734431502.58 total 108.27 last 145.73) - open/close 17129 (time 1734431512.58 total 118.28 last 141.37) - open/close 18525 (time 1734431522.58 total 128.28 last 139.55) - open/close 20000 (time 1734431531.95 total 137.64 last 157.52) - open/close 21798 (time 1734431541.96 total 147.66 last 179.58) - open/close 23291 (time 1734431551.96 total 157.66 last 149.27) - open/close 25000 (time 1734431561.98 total 167.68 last 170.61) - open/close 26804 (time 1734431571.99 total 177.68 last 180.28) - open/close 28315 (time 1734431582.02 total 187.72 last 150.50) - open/close 29861 (time 1734431592.03 total 197.73 last 154.52) - open/close 31393 (time 1734431602.04 total 207.73 last 153.12) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 215.12 seconds: 151.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3217 (time 1734431620.84 total 10.00 last 321.55) - /unlink 6137 (time 1734431630.84 total 20.01 last 291.93) - /unlink 9190 (time 1734431640.84 total 30.01 last 305.25) - /unlink 10000 (time 1734431643.76 total 32.93 last 277.30) - /unlink 12918 (time 1734431653.76 total 42.93 last 291.75) - /unlink 15824 (time 1734431663.76 total 52.93 last 290.60) - /unlink 19512 (time 1734431673.76 total 62.93 last 368.80) - /unlink 23095 (time 1734431683.76 total 72.93 last 358.29) - /unlink 26972 (time 1734431693.77 total 82.93 last 387.63) - /unlink 30000 (time 1734431703.14 total 92.31 last 322.96) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 99.99 seconds: 325.62 ops/second Waiting for MDT destroys to complete oleg205-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg205-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 - open/close 1931 (time 1734431732.83 total 10.00 last 193.08) - open/close 3893 (time 1734431742.83 total 20.00 last 196.17) - open/close 6177 (time 1734431752.84 total 30.00 last 228.38) - open/close 8062 (time 1734431762.84 total 40.01 last 188.41) - open/close 9960 (time 1734431772.84 total 50.01 last 189.75) - open/close 11704 (time 1734431782.85 total 60.01 last 174.36) - open/close 13945 (time 1734431792.85 total 70.01 last 224.10) - open/close 16291 (time 1734431802.85 total 80.01 last 234.58) - open/close 18919 (time 1734431812.85 total 90.02 last 262.75) - open/close 20000 (time 1734431818.32 total 95.49 last 197.57) - open/close 21624 (time 1734431828.32 total 105.49 last 162.34) - open/close 23848 (time 1734431838.33 total 115.49 last 222.33) - open/close 26052 (time 1734431848.33 total 125.50 last 220.34) - open/close 28061 (time 1734431858.33 total 135.50 last 200.89) - open/close 30000 (time 1734431867.89 total 145.05 last 202.90) - open/close 32038 (time 1734431877.89 total 155.06 last 203.78) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 157.39 seconds: 206.88 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4159 (time 1734431891.53 total 10.00 last 415.88) - /unlink 8256 (time 1734431901.53 total 20.00 last 409.66) - /unlink 10000 (time 1734431904.97 total 23.44 last 506.98) - /unlink 13863 (time 1734431914.97 total 33.44 last 386.27) - /unlink 17911 (time 1734431924.97 total 43.44 last 404.72) - /unlink 20000 (time 1734431930.13 total 48.60 last 404.89) - /unlink 24949 (time 1734431940.13 total 58.60 last 494.86) - /unlink 29593 (time 1734431950.13 total 68.60 last 464.40) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 75.00 seconds: 434.14 ops/second Waiting for MDT destroys to complete oleg205-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg205-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server unloading modules via unload_modules_local on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2