== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:47:58 (1734259678) Stopping clients: oleg109-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg109-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 oleg109-server: oleg109-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: detected 4 online CPUs by sysfs oleg109-server: Force libcfs to create 2 CPU partitions oleg109-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg109-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg109-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg109-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg109-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1936 (time 1734259740.89 total 10.00 last 193.56) - open/close 4033 (time 1734259750.90 total 20.00 last 209.65) - open/close 6001 (time 1734259760.90 total 30.01 last 196.68) - open/close 7909 (time 1734259770.90 total 40.01 last 190.78) - open/close 10000 (time 1734259779.80 total 48.91 last 235.01) - open/close 12242 (time 1734259789.80 total 58.91 last 224.19) - open/close 14552 (time 1734259799.80 total 68.91 last 230.94) - open/close 16681 (time 1734259809.81 total 78.91 last 212.84) - open/close 18981 (time 1734259819.81 total 88.92 last 229.99) - open/close 20000 (time 1734259824.47 total 93.58 last 218.46) - open/close 21888 (time 1734259834.48 total 103.59 last 188.68) - open/close 23696 (time 1734259844.48 total 113.59 last 180.74) - open/close 25692 (time 1734259854.48 total 123.59 last 199.52) - open/close 27629 (time 1734259864.49 total 133.60 last 193.63) - open/close 29460 (time 1734259874.49 total 143.60 last 183.05) - open/close 30000 (time 1734259877.68 total 146.79 last 169.49) - open/close 31905 (time 1734259887.68 total 156.79 last 190.49) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 160.03 seconds: 203.47 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3595 (time 1734259902.17 total 10.00 last 359.42) - /unlink 7579 (time 1734259912.18 total 20.00 last 398.31) - /unlink 10000 (time 1734259917.93 total 25.76 last 420.61) - /unlink 14282 (time 1734259927.93 total 35.76 last 428.20) - /unlink 18328 (time 1734259937.93 total 45.76 last 404.59) - /unlink 20000 (time 1734259941.71 total 49.54 last 442.32) - /unlink 24137 (time 1734259951.71 total 59.54 last 413.64) - /unlink 28771 (time 1734259961.72 total 69.54 last 463.36) - /unlink 30000 (time 1734259964.43 total 72.26 last 452.61) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 78.54 seconds: 414.59 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 - open/close 2466 (time 1734259990.81 total 10.00 last 246.57) - open/close 5007 (time 1734260000.81 total 20.00 last 254.05) - open/close 7676 (time 1734260010.81 total 30.00 last 266.89) - open/close 10000 (time 1734260020.25 total 39.45 last 246.13) - open/close 12490 (time 1734260030.25 total 49.45 last 248.94) - open/close 14911 (time 1734260040.26 total 59.45 last 242.09) - open/close 17317 (time 1734260050.26 total 69.45 last 240.56) - open/close 19598 (time 1734260060.26 total 79.45 last 228.08) - open/close 21608 (time 1734260070.26 total 89.46 last 200.89) - open/close 23674 (time 1734260080.27 total 99.46 last 206.55) - open/close 25860 (time 1734260090.27 total 109.46 last 218.60) - open/close 28002 (time 1734260100.27 total 119.46 last 214.11) - open/close 30000 (time 1734260109.49 total 128.68 last 216.74) - open/close 32244 (time 1734260119.49 total 138.68 last 224.37) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 140.07 seconds: 232.45 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4380 (time 1734260132.08 total 10.00 last 437.85) - /unlink 8859 (time 1734260142.08 total 20.00 last 447.88) - /unlink 10000 (time 1734260144.64 total 22.56 last 445.91) - /unlink 14705 (time 1734260154.64 total 32.56 last 470.43) - /unlink 20000 (time 1734260164.22 total 42.14 last 553.20) - /unlink 25090 (time 1734260174.22 total 52.14 last 508.92) - /unlink 30000 (time 1734260182.44 total 60.36 last 597.21) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 64.44 seconds: 505.29 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 - open/close 3172 (time 1734260204.53 total 10.00 last 317.11) - open/close 6359 (time 1734260214.53 total 20.01 last 318.62) - open/close 9644 (time 1734260224.53 total 30.01 last 328.48) - open/close 13103 (time 1734260234.53 total 40.01 last 345.85) - open/close 16217 (time 1734260244.53 total 50.01 last 311.38) - open/close 19436 (time 1734260254.53 total 60.01 last 321.85) - open/close 22787 (time 1734260264.53 total 70.01 last 335.05) - open/close 26137 (time 1734260274.54 total 80.01 last 334.91) - open/close 29772 (time 1734260284.54 total 90.01 last 363.46) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 97.93 seconds: 332.49 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5919 (time 1734260303.46 total 10.00 last 591.86) - /unlink 10000 (time 1734260310.93 total 17.47 last 546.42) - /unlink 14781 (time 1734260320.93 total 27.47 last 478.09) - /unlink 20000 (time 1734260330.85 total 37.40 last 525.72) - /unlink 25109 (time 1734260340.86 total 47.40 last 510.83) - /unlink 30000 (time 1734260349.56 total 56.11 last 561.73) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 60.53 seconds: 537.88 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 - open/close 3368 (time 1734260373.96 total 10.00 last 336.80) - open/close 6622 (time 1734260383.96 total 20.00 last 325.35) - open/close 10000 (time 1734260393.78 total 29.81 last 344.28) - open/close 13614 (time 1734260403.78 total 39.81 last 361.37) - open/close 17031 (time 1734260413.78 total 49.82 last 341.54) - open/close 19951 (time 1734260423.79 total 59.82 last 291.91) - open/close 23193 (time 1734260433.79 total 69.82 last 324.16) - open/close 25848 (time 1734260443.79 total 79.83 last 265.38) - open/close 28372 (time 1734260453.79 total 89.83 last 252.31) - open/close 30000 (time 1734260459.83 total 95.87 last 269.66) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 104.98 seconds: 310.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32808 47184 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33063 33063 0 100% /mnt/lustre - /unlink 4617 (time 1734260480.08 total 10.00 last 461.69) - /unlink 9636 (time 1734260490.10 total 20.02 last 500.92) - /unlink 14448 (time 1734260500.10 total 30.02 last 481.16) - /unlink 20000 (time 1734260509.82 total 39.74 last 570.99) - /unlink 26402 (time 1734260519.82 total 49.75 last 640.11) - /unlink 30000 (time 1734260525.05 total 54.97 last 688.87) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 58.75 seconds: 554.18 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 - open/close 3354 (time 1734260542.88 total 10.00 last 335.30) - open/close 6717 (time 1734260552.88 total 20.00 last 336.29) - open/close 9940 (time 1734260562.88 total 30.00 last 322.26) - open/close 13951 (time 1734260572.88 total 40.01 last 401.07) - open/close 18084 (time 1734260582.88 total 50.01 last 413.25) - open/close 20000 (time 1734260587.45 total 54.58 last 419.16) - open/close 23341 (time 1734260597.45 total 64.58 last 334.01) - open/close 27160 (time 1734260607.45 total 74.58 last 381.89) - open/close 30000 (time 1734260613.75 total 80.88 last 450.61) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 86.79 seconds: 375.14 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7498 (time 1734260630.70 total 10.00 last 749.74) - /unlink 10000 (time 1734260633.34 total 12.65 last 945.41) - /unlink 19172 (time 1734260643.34 total 22.65 last 917.19) - /unlink 25197 (time 1734260653.34 total 32.65 last 602.46) - /unlink 30000 (time 1734260660.87 total 40.18 last 637.90) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 44.25 seconds: 735.86 ops/second Waiting for MDT destroys to complete oleg109-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 oleg109-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg109-server unloading modules via unload_modules_local on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2