-----============= acceptance-small: conf-sanity ============----- Tue Nov 5 10:48:45 EST 2024 excepting tests: 32newtarball 110 Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg231-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg231-server oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg231-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg231-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg231-server: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg231-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:50:12 (1730821812) Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg231-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock oleg231-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1648 (time 1730821863.77 total 10.00 last 164.73) - open/close 3329 (time 1730821873.77 total 20.00 last 168.09) - open/close 5193 (time 1730821883.77 total 30.01 last 186.34) - open/close 6746 (time 1730821893.77 total 40.01 last 155.29) - open/close 8404 (time 1730821903.77 total 50.01 last 165.77) - open/close 10000 (time 1730821912.20 total 58.44 last 189.33) - open/close 12220 (time 1730821922.20 total 68.44 last 221.98) - open/close 13937 (time 1730821932.21 total 78.44 last 171.65) - open/close 15504 (time 1730821942.21 total 88.45 last 156.66) - open/close 17484 (time 1730821952.21 total 98.45 last 197.97) - open/close 18857 (time 1730821962.22 total 108.45 last 137.22) - open/close 20000 (time 1730821969.49 total 115.73 last 157.20) - open/close 21750 (time 1730821979.49 total 125.73 last 174.96) - open/close 23716 (time 1730821989.50 total 135.73 last 196.51) - open/close 25527 (time 1730821999.50 total 145.74 last 180.99) - open/close 27623 (time 1730822009.50 total 155.74 last 209.57) - open/close 29399 (time 1730822019.51 total 165.74 last 177.56) - open/close 30000 (time 1730822022.48 total 168.71 last 202.39) - open/close 32124 (time 1730822032.48 total 178.71 last 212.38) - open/close 34567 (time 1730822042.48 total 188.71 last 244.29) - open/close 36877 (time 1730822052.48 total 198.71 last 230.99) - open/close 39217 (time 1730822062.48 total 208.71 last 233.99) - open/close 40000 (time 1730822065.32 total 211.55 last 275.84) - open/close 42382 (time 1730822075.32 total 221.56 last 238.13) - open/close 44764 (time 1730822085.32 total 231.56 last 238.14) - open/close 47171 (time 1730822095.32 total 241.56 last 240.65) - open/close 50000 (time 1730822105.20 total 251.43 last 286.56) - open/close 52721 (time 1730822115.20 total 261.43 last 272.10) - open/close 55444 (time 1730822125.20 total 271.43 last 272.27) - open/close 58488 (time 1730822135.20 total 281.43 last 304.35) - open/close 60000 (time 1730822139.48 total 285.72 last 352.80) - open/close 63135 (time 1730822149.49 total 295.72 last 313.42) total: 64768 open/close in 301.27 seconds: 214.98 ops/second - /unlink 5908 (time 1730822166.01 total 10.00 last 590.78) - /unlink 10000 (time 1730822172.37 total 16.36 last 643.20) - /unlink 16292 (time 1730822182.37 total 26.36 last 629.15) - /unlink 20000 (time 1730822188.47 total 32.45 last 608.76) - /unlink 25845 (time 1730822198.47 total 42.45 last 584.46) - /unlink 30000 (time 1730822204.20 total 48.19 last 724.56) - /unlink 38516 (time 1730822214.20 total 58.19 last 851.51) - /unlink 47354 (time 1730822224.21 total 68.19 last 883.52) - /unlink 50000 (time 1730822226.96 total 70.95 last 960.29) - /unlink 59041 (time 1730822236.96 total 80.95 last 904.04) total: 64768 /unlink in 87.39 seconds: 741.17 ops/second - open/close 5010 (time 1730822254.29 total 10.00 last 500.99) - open/close 10000 (time 1730822263.58 total 19.29 last 537.15) - open/close 15302 (time 1730822273.59 total 29.29 last 530.11) - open/close 20000 (time 1730822283.06 total 38.77 last 495.65) - open/close 25433 (time 1730822293.06 total 48.77 last 543.30) - open/close 30000 (time 1730822302.21 total 57.92 last 499.26) - open/close 34906 (time 1730822312.21 total 67.92 last 490.54) - open/close 40000 (time 1730822321.54 total 77.24 last 546.29) - open/close 45047 (time 1730822331.54 total 87.25 last 504.53) - open/close 49490 (time 1730822341.54 total 97.25 last 444.29) - open/close 54062 (time 1730822351.54 total 107.25 last 457.13) - open/close 59341 (time 1730822361.54 total 117.25 last 527.88) total: 64768 open/close in 127.07 seconds: 509.69 ops/second - /unlink 10000 (time 1730822381.80 total 9.62 last 1039.20) - /unlink 18432 (time 1730822391.80 total 19.62 last 843.14) - /unlink 28363 (time 1730822401.80 total 29.62 last 993.08) - /unlink 39054 (time 1730822411.80 total 39.62 last 1069.02) - /unlink 48128 (time 1730822421.80 total 49.63 last 907.34) - /unlink 50000 (time 1730822423.94 total 51.76 last 875.06) - /unlink 58972 (time 1730822433.94 total 61.76 last 897.19) total: 64768 /unlink in 68.11 seconds: 950.90 ops/second - open/close 5387 (time 1730822451.16 total 10.00 last 538.67) - open/close 10000 (time 1730822460.07 total 18.91 last 517.65) - open/close 15231 (time 1730822470.07 total 28.91 last 523.03) - open/close 20000 (time 1730822479.17 total 38.01 last 524.20) - open/close 25292 (time 1730822489.17 total 48.01 last 529.12) - open/close 30000 (time 1730822498.20 total 57.04 last 521.66) - open/close 35209 (time 1730822508.20 total 67.04 last 520.85) - open/close 40000 (time 1730822517.84 total 76.68 last 496.93) - open/close 44903 (time 1730822527.84 total 86.68 last 490.27) - open/close 49740 (time 1730822537.84 total 96.68 last 483.70) - open/close 54486 (time 1730822547.84 total 106.68 last 474.60) - open/close 59375 (time 1730822557.84 total 116.68 last 488.87) - open/close 64423 (time 1730822567.84 total 126.68 last 504.74) total: 64768 open/close in 127.41 seconds: 508.35 ops/second - /unlink 9309 (time 1730822579.43 total 10.00 last 930.82) - /unlink 18551 (time 1730822589.43 total 20.00 last 924.19) - /unlink 27925 (time 1730822599.43 total 30.00 last 937.39) - /unlink 30000 (time 1730822601.64 total 32.21 last 940.63) - /unlink 39329 (time 1730822611.64 total 42.21 last 932.82) - /unlink 48528 (time 1730822621.64 total 52.21 last 919.86) - /unlink 58061 (time 1730822631.64 total 62.21 last 953.27) - /unlink 60000 (time 1730822633.73 total 64.30 last 926.28) total: 64768 /unlink in 69.35 seconds: 933.88 ops/second - open/close 4930 (time 1730822649.84 total 10.00 last 492.97) - open/close 10000 (time 1730822659.60 total 19.76 last 519.34) - open/close 15257 (time 1730822669.60 total 29.76 last 525.66) - open/close 20000 (time 1730822678.61 total 38.77 last 526.41) - open/close 24867 (time 1730822688.62 total 48.78 last 486.65) - open/close 29999 (time 1730822698.62 total 58.78 last 513.12) - open/close 35107 (time 1730822708.62 total 68.78 last 510.71) - open/close 40000 (time 1730822718.54 total 78.70 last 493.31) - open/close 45216 (time 1730822728.54 total 88.70 last 521.54) - open/close 50000 (time 1730822737.90 total 98.06 last 511.26) - open/close 55335 (time 1730822747.90 total 108.06 last 533.47) - open/close 60000 (time 1730822756.83 total 116.99 last 522.11) total: 64768 open/close in 126.10 seconds: 513.63 ops/second - /unlink 9344 (time 1730822776.80 total 10.00 last 934.39) - /unlink 18631 (time 1730822786.80 total 20.00 last 928.70) - /unlink 27595 (time 1730822796.80 total 30.00 last 896.31) - /unlink 30000 (time 1730822799.40 total 32.60 last 924.98) - /unlink 38930 (time 1730822809.40 total 42.60 last 892.98) - /unlink 48115 (time 1730822819.40 total 52.60 last 918.47) - /unlink 50000 (time 1730822821.48 total 54.69 last 904.30) - /unlink 59278 (time 1730822831.48 total 64.69 last 927.77) total: 64768 /unlink in 70.38 seconds: 920.28 ops/second - open/close 5151 (time 1730822848.00 total 10.00 last 515.07) - open/close 10000 (time 1730822856.87 total 18.87 last 546.49) - open/close 15420 (time 1730822866.87 total 28.87 last 541.95) - open/close 20000 (time 1730822875.42 total 37.42 last 536.00) - open/close 25314 (time 1730822885.42 total 47.42 last 531.35) - open/close 30000 (time 1730822894.07 total 56.07 last 541.92) - open/close 35298 (time 1730822904.07 total 66.07 last 529.72) - open/close 40000 (time 1730822913.07 total 75.07 last 522.52) - open/close 44958 (time 1730822923.07 total 85.07 last 495.76) - open/close 49759 (time 1730822933.07 total 95.07 last 480.05) - open/close 54793 (time 1730822943.07 total 105.07 last 503.31) - open/close 59399 (time 1730822953.07 total 115.07 last 460.52) - open/close 64259 (time 1730822963.07 total 125.07 last 485.92) total: 64768 open/close in 126.09 seconds: 513.67 ops/second - /unlink 9351 (time 1730822974.92 total 10.00 last 935.02) - /unlink 18611 (time 1730822984.92 total 20.00 last 925.98) - /unlink 27129 (time 1730822994.92 total 30.00 last 851.76) - /unlink 30000 (time 1730822997.93 total 33.01 last 952.94) - /unlink 39340 (time 1730823007.93 total 43.02 last 933.93) - /unlink 48784 (time 1730823017.93 total 53.02 last 944.35) - /unlink 58085 (time 1730823027.93 total 63.02 last 930.08) total: 64768 /unlink in 69.83 seconds: 927.49 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server LNET ready to unload unloading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local oleg231-server: LNET ready to unload modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 PASS 106 (1268s) Stopping clients: oleg231-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg231-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 oleg231-server: oleg231-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1373 sec ========== 11:11:39 (1730823099)