== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:48:31 (1734259711) Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 oleg306-server: oleg306-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg306-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg306-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2343 (time 1734259773.92 total 10.00 last 234.20) - open/close 4776 (time 1734259783.93 total 20.01 last 243.25) - open/close 7260 (time 1734259793.93 total 30.01 last 248.33) - open/close 9408 (time 1734259803.93 total 40.01 last 214.71) - open/close 10000 (time 1734259806.71 total 42.79 last 213.42) - open/close 12144 (time 1734259816.71 total 52.79 last 214.31) - open/close 14395 (time 1734259826.71 total 62.79 last 225.06) - open/close 16941 (time 1734259836.72 total 72.80 last 254.56) - open/close 19539 (time 1734259846.72 total 82.80 last 259.79) - open/close 22141 (time 1734259856.72 total 92.80 last 260.13) - open/close 24672 (time 1734259866.72 total 102.80 last 253.02) - open/close 26906 (time 1734259876.72 total 112.80 last 223.34) - open/close 29049 (time 1734259886.73 total 122.81 last 214.20) - open/close 30000 (time 1734259891.20 total 127.28 last 212.68) - open/close 32012 (time 1734259901.21 total 137.29 last 201.06) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 139.94 seconds: 232.67 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4097 (time 1734259915.06 total 10.00 last 409.62) - /unlink 8650 (time 1734259925.06 total 20.00 last 455.30) - /unlink 10000 (time 1734259928.06 total 23.00 last 450.02) - /unlink 14400 (time 1734259938.06 total 33.00 last 439.98) - /unlink 18534 (time 1734259948.07 total 43.00 last 413.39) - /unlink 20000 (time 1734259951.64 total 46.58 last 410.13) - /unlink 24401 (time 1734259961.64 total 56.58 last 440.06) - /unlink 28590 (time 1734259971.64 total 66.58 last 418.83) - /unlink 30000 (time 1734259974.81 total 69.74 last 445.65) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 75.68 seconds: 430.24 ops/second Waiting for MDT destroys to complete oleg306-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg306-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 - open/close 2733 (time 1734259998.09 total 10.00 last 273.29) - open/close 5977 (time 1734260008.09 total 20.00 last 324.38) - open/close 8958 (time 1734260018.09 total 30.00 last 298.07) - open/close 10000 (time 1734260021.68 total 33.60 last 289.94) - open/close 12855 (time 1734260031.68 total 43.60 last 285.46) - open/close 15708 (time 1734260041.69 total 53.60 last 285.26) - open/close 18478 (time 1734260051.69 total 63.60 last 276.99) - open/close 20000 (time 1734260056.61 total 68.52 last 309.02) - open/close 23733 (time 1734260066.61 total 78.53 last 373.18) - open/close 26780 (time 1734260076.61 total 88.53 last 304.69) - open/close 29796 (time 1734260086.62 total 98.53 last 301.52) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 108.04 seconds: 301.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5505 (time 1734260107.16 total 10.00 last 550.36) - /unlink 10000 (time 1734260114.98 total 17.82 last 575.32) - /unlink 15252 (time 1734260124.98 total 27.82 last 525.15) - /unlink 20000 (time 1734260134.26 total 37.09 last 511.75) - /unlink 25177 (time 1734260144.26 total 47.10 last 517.58) - /unlink 30000 (time 1734260152.64 total 55.48 last 575.63) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 59.63 seconds: 545.99 ops/second Waiting for MDT destroys to complete oleg306-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg306-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 - open/close 3260 (time 1734260174.75 total 10.00 last 325.94) - open/close 6527 (time 1734260184.75 total 20.00 last 326.69) - open/close 9933 (time 1734260194.75 total 30.00 last 340.55) - open/close 12927 (time 1734260204.75 total 40.01 last 299.35) - open/close 16020 (time 1734260214.75 total 50.01 last 309.29) - open/close 18972 (time 1734260224.75 total 60.01 last 295.14) - open/close 20000 (time 1734260228.45 total 63.71 last 277.81) - open/close 22782 (time 1734260238.45 total 73.71 last 278.18) - open/close 25558 (time 1734260248.46 total 83.71 last 277.54) - open/close 28865 (time 1734260258.46 total 93.71 last 330.65) - open/close 30000 (time 1734260261.74 total 96.99 last 346.19) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 105.34 seconds: 309.10 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4890 (time 1734260281.16 total 10.00 last 488.96) - /unlink 10000 (time 1734260290.88 total 19.72 last 525.72) - /unlink 14507 (time 1734260300.88 total 29.72 last 450.69) - /unlink 19384 (time 1734260310.88 total 39.72 last 487.70) - /unlink 24359 (time 1734260320.88 total 49.72 last 497.46) - /unlink 29893 (time 1734260330.88 total 59.72 last 553.33) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 64.74 seconds: 502.91 ops/second Waiting for MDT destroys to complete oleg306-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 oleg306-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 - open/close 3171 (time 1734260354.13 total 10.00 last 317.00) - open/close 5931 (time 1734260364.13 total 20.01 last 275.90) - open/close 8847 (time 1734260374.14 total 30.01 last 291.54) - open/close 10000 (time 1734260378.53 total 34.40 last 262.60) - open/close 12626 (time 1734260388.53 total 44.40 last 262.57) - open/close 15195 (time 1734260398.53 total 54.40 last 256.84) - open/close 17614 (time 1734260408.53 total 64.41 last 241.83) - open/close 20000 (time 1734260417.88 total 73.75 last 255.38) - open/close 22707 (time 1734260427.88 total 83.75 last 270.59) - open/close 25329 (time 1734260437.89 total 93.76 last 262.08) - open/close 27705 (time 1734260447.89 total 103.76 last 237.54) - open/close 30000 (time 1734260456.78 total 112.65 last 258.17) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 122.04 seconds: 266.79 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5537 (time 1734260477.21 total 10.00 last 553.70) - /unlink 10000 (time 1734260484.29 total 17.08 last 630.29) - /unlink 15676 (time 1734260494.30 total 27.08 last 567.41) - /unlink 20000 (time 1734260501.49 total 34.28 last 600.74) - /unlink 26338 (time 1734260511.50 total 44.28 last 633.73) - /unlink 30000 (time 1734260517.37 total 50.16 last 623.52) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 54.23 seconds: 600.45 ops/second Waiting for MDT destroys to complete oleg306-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg306-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 - open/close 3433 (time 1734260543.30 total 10.00 last 343.21) - open/close 6142 (time 1734260553.30 total 20.01 last 270.80) - open/close 9141 (time 1734260563.30 total 30.01 last 299.85) - open/close 10000 (time 1734260566.27 total 32.97 last 289.62) - open/close 13350 (time 1734260576.27 total 42.98 last 334.95) - open/close 16605 (time 1734260586.27 total 52.98 last 325.41) - open/close 19774 (time 1734260596.28 total 62.98 last 316.83) - open/close 22774 (time 1734260606.28 total 72.98 last 299.90) - open/close 25767 (time 1734260616.28 total 82.99 last 299.22) - open/close 28926 (time 1734260626.28 total 92.99 last 315.88) - open/close 30000 (time 1734260629.93 total 96.63 last 294.41) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 105.64 seconds: 308.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5848 (time 1734260649.97 total 10.00 last 584.71) - /unlink 10000 (time 1734260656.99 total 17.02 last 591.79) - /unlink 16797 (time 1734260666.99 total 27.02 last 679.70) - /unlink 20000 (time 1734260672.59 total 32.62 last 571.87) - /unlink 25826 (time 1734260682.59 total 42.62 last 582.58) - /unlink 30000 (time 1734260689.28 total 49.31 last 623.68) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 53.39 seconds: 609.90 ops/second Waiting for MDT destroys to complete oleg306-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg306-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg306-server unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2