== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:56:17 (1735098977) Stopping clients: oleg458-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg458-client.virtnet /mnt/lustre2 (opts:-f) oleg458-server: oleg458-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: detected 4 online CPUs by sysfs oleg458-server: MODOPTS_LIBCFS= oleg458-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg458-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg458-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg458-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2525 (time 1735099024.10 total 10.00 last 252.40) - open/close 5568 (time 1735099034.10 total 20.01 last 304.26) - open/close 8962 (time 1735099044.11 total 30.01 last 339.31) - open/close 10000 (time 1735099047.76 total 33.66 last 284.40) - open/close 12527 (time 1735099057.76 total 43.66 last 252.67) - open/close 15311 (time 1735099067.76 total 53.66 last 278.35) - open/close 19194 (time 1735099077.76 total 63.66 last 388.16) - open/close 20000 (time 1735099079.94 total 65.84 last 371.17) - open/close 23505 (time 1735099089.94 total 75.84 last 350.46) - open/close 26601 (time 1735099099.94 total 85.84 last 309.52) - open/close 29072 (time 1735099109.94 total 95.84 last 247.06) - open/close 30000 (time 1735099112.62 total 98.52 last 346.18) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 105.92 seconds: 307.40 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7832 (time 1735099131.03 total 10.00 last 783.19) - /unlink 10000 (time 1735099133.61 total 12.58 last 839.34) - /unlink 18806 (time 1735099143.61 total 22.58 last 880.51) - /unlink 27228 (time 1735099153.61 total 32.59 last 842.10) - /unlink 30000 (time 1735099157.24 total 36.21 last 763.83) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 39.63 seconds: 821.62 ops/second Waiting for MDT destroys to complete oleg458-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg458-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 - open/close 3185 (time 1735099196.68 total 10.00 last 318.49) - open/close 6515 (time 1735099206.68 total 20.00 last 332.99) - open/close 9957 (time 1735099216.68 total 30.00 last 344.18) - open/close 13212 (time 1735099226.68 total 40.00 last 325.41) - open/close 16626 (time 1735099236.68 total 50.01 last 341.36) - open/close 19929 (time 1735099246.68 total 60.01 last 330.28) - open/close 23413 (time 1735099256.68 total 70.01 last 348.38) - open/close 26869 (time 1735099266.68 total 80.01 last 345.58) - open/close 30000 (time 1735099276.38 total 89.70 last 322.92) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 97.91 seconds: 332.55 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6848 (time 1735099295.61 total 10.00 last 684.73) - /unlink 10000 (time 1735099299.95 total 14.34 last 726.94) - /unlink 17424 (time 1735099309.95 total 24.34 last 742.39) - /unlink 20000 (time 1735099313.19 total 27.59 last 792.75) - /unlink 26688 (time 1735099323.20 total 37.59 last 668.77) - /unlink 30000 (time 1735099329.24 total 43.63 last 547.76) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 49.39 seconds: 659.22 ops/second Waiting for MDT destroys to complete oleg458-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 oleg458-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 - open/close 1668 (time 1735099378.62 total 10.00 last 166.78) - open/close 3646 (time 1735099388.62 total 20.00 last 197.76) - open/close 5488 (time 1735099398.63 total 30.01 last 184.13) - open/close 7255 (time 1735099408.63 total 40.01 last 176.61) - open/close 9302 (time 1735099418.64 total 50.02 last 204.64) - open/close 10000 (time 1735099421.35 total 52.73 last 257.16) - open/close 13155 (time 1735099431.35 total 62.73 last 315.40) - open/close 16604 (time 1735099441.36 total 72.73 last 344.86) - open/close 19203 (time 1735099451.36 total 82.74 last 259.84) - open/close 20000 (time 1735099453.87 total 85.25 last 317.40) - open/close 22368 (time 1735099463.87 total 95.25 last 236.76) - open/close 24948 (time 1735099473.87 total 105.25 last 257.94) - open/close 27543 (time 1735099483.88 total 115.25 last 259.41) - open/close 29953 (time 1735099493.88 total 125.26 last 240.97) - open/close 32261 (time 1735099503.88 total 135.26 last 230.77) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 136.51 seconds: 238.51 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4282 (time 1735099516.17 total 10.00 last 428.14) - /unlink 8700 (time 1735099526.18 total 20.00 last 441.75) - /unlink 10000 (time 1735099529.27 total 23.09 last 420.47) - /unlink 14429 (time 1735099539.27 total 33.09 last 442.89) - /unlink 19013 (time 1735099549.27 total 43.10 last 458.35) - /unlink 20000 (time 1735099551.56 total 45.39 last 430.06) - /unlink 24715 (time 1735099561.56 total 55.39 last 471.50) - /unlink 28692 (time 1735099571.56 total 65.39 last 397.68) - /unlink 30000 (time 1735099575.00 total 68.83 last 380.84) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 75.56 seconds: 430.92 ops/second Waiting for MDT destroys to complete oleg458-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg458-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 - open/close 2008 (time 1735099603.49 total 10.00 last 200.72) - open/close 4080 (time 1735099613.50 total 20.01 last 207.13) - open/close 6363 (time 1735099623.50 total 30.01 last 228.26) - open/close 8238 (time 1735099633.50 total 40.01 last 187.40) - open/close 10000 (time 1735099642.75 total 49.27 last 190.44) - open/close 11947 (time 1735099652.76 total 59.27 last 194.67) - open/close 14274 (time 1735099662.76 total 69.27 last 232.70) - open/close 16319 (time 1735099672.76 total 79.27 last 204.45) - open/close 18081 (time 1735099682.77 total 89.28 last 176.08) - open/close 19880 (time 1735099692.77 total 99.28 last 179.84) - open/close 21808 (time 1735099702.77 total 109.28 last 192.74) - open/close 23740 (time 1735099712.77 total 119.28 last 193.17) - open/close 25655 (time 1735099722.77 total 129.29 last 191.46) - open/close 27468 (time 1735099732.78 total 139.29 last 181.23) - open/close 29536 (time 1735099742.78 total 149.29 last 206.76) - open/close 30000 (time 1735099745.06 total 151.57 last 203.73) - open/close 32147 (time 1735099755.06 total 161.57 last 214.63) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 163.34 seconds: 199.34 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4866 (time 1735099767.94 total 10.00 last 486.52) - /unlink 9503 (time 1735099777.94 total 20.00 last 463.69) - /unlink 13924 (time 1735099787.94 total 30.00 last 442.07) - /unlink 18867 (time 1735099797.94 total 40.01 last 494.17) - /unlink 20000 (time 1735099800.02 total 42.08 last 545.57) - /unlink 24759 (time 1735099810.02 total 52.08 last 475.85) - /unlink 29492 (time 1735099820.02 total 62.08 last 473.23) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 68.00 seconds: 478.79 ops/second Waiting for MDT destroys to complete oleg458-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg458-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 - open/close 2959 (time 1735099848.23 total 10.00 last 295.88) - open/close 5963 (time 1735099858.23 total 20.00 last 300.33) - open/close 8654 (time 1735099868.23 total 30.00 last 269.06) - open/close 10000 (time 1735099873.28 total 35.05 last 266.95) - open/close 12624 (time 1735099883.28 total 45.05 last 262.33) - open/close 15194 (time 1735099893.28 total 55.05 last 256.95) - open/close 17468 (time 1735099903.28 total 65.05 last 227.36) - open/close 19792 (time 1735099913.29 total 75.06 last 232.32) - open/close 21780 (time 1735099923.29 total 85.06 last 198.71) - open/close 24060 (time 1735099933.29 total 95.06 last 227.99) - open/close 26332 (time 1735099943.29 total 105.06 last 227.19) - open/close 28137 (time 1735099953.30 total 115.07 last 180.43) - open/close 29801 (time 1735099963.30 total 125.07 last 166.37) - open/close 31495 (time 1735099973.30 total 135.07 last 169.28) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 139.77 seconds: 232.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4429 (time 1735099989.08 total 10.00 last 442.88) - /unlink 7041 (time 1735099999.09 total 20.01 last 261.04) - /unlink 8787 (time 1735100009.09 total 30.01 last 174.57) - /unlink 10000 (time 1735100016.41 total 37.32 last 165.82) - /unlink 12501 (time 1735100026.41 total 47.32 last 250.07) - /unlink 15052 (time 1735100036.41 total 57.33 last 255.02) - /unlink 17812 (time 1735100046.41 total 67.33 last 275.98) - /unlink 20000 (time 1735100053.87 total 74.78 last 293.46) - /unlink 22970 (time 1735100063.87 total 84.79 last 296.93) - /unlink 25676 (time 1735100073.87 total 94.79 last 270.56) - /unlink 28499 (time 1735100083.87 total 104.79 last 282.24) - /unlink 30000 (time 1735100089.06 total 109.98 last 289.16) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 118.96 seconds: 273.72 ops/second Waiting for MDT destroys to complete oleg458-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg458-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg458-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg458-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server stop mds service on oleg458-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server stop mds service on oleg458-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg458-server unloading modules via unload_modules_local on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2