== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:25:21 (1734258321) Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: oleg447-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4642 (time 1734258367.04 total 10.00 last 464.09) - open/close 8257 (time 1734258377.04 total 20.00 last 361.43) - open/close 10000 (time 1734258381.17 total 24.14 last 421.44) - open/close 13890 (time 1734258391.18 total 34.14 last 388.94) - open/close 17949 (time 1734258401.18 total 44.14 last 405.86) - open/close 20000 (time 1734258405.74 total 48.70 last 449.87) - open/close 24195 (time 1734258415.74 total 58.70 last 419.44) - open/close 27642 (time 1734258425.74 total 68.70 last 344.65) - open/close 30000 (time 1734258431.78 total 74.74 last 390.64) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 82.77 seconds: 394.19 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4821 (time 1734258450.95 total 10.00 last 482.09) - /unlink 9895 (time 1734258460.95 total 20.00 last 507.40) - /unlink 16177 (time 1734258470.95 total 30.00 last 628.19) - /unlink 20000 (time 1734258476.00 total 35.04 last 758.20) - /unlink 26610 (time 1734258486.00 total 45.04 last 660.94) - /unlink 30000 (time 1734258490.80 total 49.85 last 705.51) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 54.65 seconds: 597.00 ops/second Waiting for MDT destroys to complete oleg447-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg447-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 - open/close 3130 (time 1734258524.07 total 10.00 last 312.93) - open/close 6139 (time 1734258534.08 total 20.00 last 300.86) - open/close 9198 (time 1734258544.08 total 30.00 last 305.86) - open/close 10000 (time 1734258546.61 total 32.54 last 316.40) - open/close 12871 (time 1734258556.61 total 42.54 last 287.07) - open/close 15952 (time 1734258566.61 total 52.54 last 308.10) - open/close 20000 (time 1734258575.44 total 61.37 last 458.31) - open/close 23879 (time 1734258585.45 total 71.37 last 387.85) - open/close 27158 (time 1734258595.45 total 81.38 last 327.78) - open/close 30000 (time 1734258604.98 total 90.91 last 298.19) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 100.55 seconds: 324.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5352 (time 1734258625.82 total 10.00 last 535.16) - /unlink 10000 (time 1734258632.10 total 16.28 last 740.06) - /unlink 16872 (time 1734258642.10 total 26.28 last 687.07) - /unlink 20000 (time 1734258647.71 total 31.89 last 557.67) - /unlink 26754 (time 1734258657.71 total 41.89 last 675.38) - /unlink 30000 (time 1734258662.63 total 46.81 last 660.59) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 50.31 seconds: 648.54 ops/second Waiting for MDT destroys to complete oleg447-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg447-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 - open/close 3150 (time 1734258693.07 total 10.00 last 315.00) - open/close 6273 (time 1734258703.07 total 20.00 last 312.27) - open/close 8827 (time 1734258713.07 total 30.00 last 255.39) - open/close 10000 (time 1734258717.25 total 34.19 last 280.35) - open/close 13201 (time 1734258727.26 total 44.19 last 320.00) - open/close 16868 (time 1734258737.26 total 54.19 last 366.62) - open/close 20000 (time 1734258745.24 total 62.17 last 392.39) - open/close 23905 (time 1734258755.24 total 72.17 last 390.44) - open/close 28015 (time 1734258765.24 total 82.18 last 410.94) - open/close 30000 (time 1734258769.81 total 86.75 last 434.39) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 94.72 seconds: 344.44 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 7762 (time 1734258788.75 total 10.00 last 775.95) - /unlink 10000 (time 1734258792.37 total 13.62 last 618.99) - /unlink 16123 (time 1734258802.37 total 23.62 last 612.25) - /unlink 20000 (time 1734258808.00 total 29.25 last 689.09) - /unlink 26036 (time 1734258818.00 total 39.25 last 603.53) - /unlink 30000 (time 1734258824.05 total 45.30 last 654.83) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 48.44 seconds: 673.53 ops/second Waiting for MDT destroys to complete oleg447-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg447-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 - open/close 2864 (time 1734258855.00 total 10.00 last 286.36) - open/close 6052 (time 1734258865.00 total 20.00 last 318.78) - open/close 9779 (time 1734258875.01 total 30.00 last 372.63) - open/close 13625 (time 1734258885.01 total 40.01 last 384.51) - open/close 17309 (time 1734258895.01 total 50.01 last 368.30) - open/close 20000 (time 1734258903.09 total 58.09 last 332.97) - open/close 23954 (time 1734258913.10 total 68.09 last 395.30) - open/close 27337 (time 1734258923.10 total 78.10 last 338.23) - open/close 30000 (time 1734258928.66 total 83.66 last 478.83) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 88.67 seconds: 367.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32878 18 100% /mnt/lustre[OST:0] filesystem_summary: 32917 32899 18 100% /mnt/lustre - /unlink 8699 (time 1734258944.53 total 10.00 last 869.85) - /unlink 16868 (time 1734258954.53 total 20.00 last 816.84) - /unlink 20000 (time 1734258958.52 total 23.99 last 784.42) - /unlink 27661 (time 1734258968.52 total 33.99 last 766.09) - /unlink 30000 (time 1734258971.25 total 36.72 last 857.88) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 40.02 seconds: 815.31 ops/second Waiting for MDT destroys to complete oleg447-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg447-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 - open/close 3589 (time 1734258998.78 total 10.00 last 358.86) - open/close 7486 (time 1734259008.78 total 20.00 last 389.65) - open/close 10000 (time 1734259014.89 total 26.10 last 412.01) - open/close 13560 (time 1734259024.89 total 36.11 last 355.94) - open/close 17379 (time 1734259034.89 total 46.11 last 381.82) - open/close 20000 (time 1734259042.34 total 53.56 last 351.70) - open/close 23482 (time 1734259052.35 total 63.56 last 348.12) - open/close 26730 (time 1734259062.35 total 73.57 last 324.70) - open/close 30000 (time 1734259070.41 total 81.63 last 405.63) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 87.66 seconds: 372.20 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 7866 (time 1734259087.37 total 10.00 last 786.32) - /unlink 10000 (time 1734259090.45 total 13.08 last 693.11) - /unlink 16435 (time 1734259100.45 total 23.08 last 643.44) - /unlink 20000 (time 1734259104.99 total 27.63 last 784.12) - /unlink 28678 (time 1734259114.99 total 37.63 last 867.77) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 42.38 seconds: 769.92 ops/second Waiting for MDT destroys to complete oleg447-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg447-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server unloading modules via unload_modules_local on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2