-----============= acceptance-small: conf-sanity ============----- Sun Dec 8 02:14:14 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg111-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg111-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg111-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg111-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg111-server oleg111-server: oleg111-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: detected 4 online CPUs by sysfs oleg111-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg111-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg111-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg111-server: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg111-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:15:10 (1733642110) Stopping clients: oleg111-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg111-client.virtnet /mnt/lustre2 (opts:-f) oleg111-server: oleg111-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: detected 4 online CPUs by sysfs oleg111-server: Force libcfs to create 2 CPU partitions oleg111-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg111-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg111-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg111-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5564 (time 1733642143.93 total 10.00 last 556.32) - open/close 10000 (time 1733642152.44 total 18.51 last 521.39) - open/close 15717 (time 1733642162.44 total 28.51 last 571.64) - open/close 20000 (time 1733642170.66 total 36.73 last 521.13) - open/close 25262 (time 1733642180.66 total 46.73 last 526.11) - open/close 30000 (time 1733642190.56 total 56.63 last 478.55) - open/close 34575 (time 1733642200.56 total 66.63 last 457.46) - open/close 39903 (time 1733642210.56 total 76.63 last 532.79) - open/close 45418 (time 1733642220.56 total 86.63 last 551.42) - open/close 50000 (time 1733642228.28 total 94.36 last 593.47) - open/close 55495 (time 1733642238.28 total 104.36 last 549.42) - open/close 60000 (time 1733642246.46 total 112.53 last 551.15) total: 64768 open/close in 121.07 seconds: 534.95 ops/second - /unlink 8968 (time 1733642265.83 total 10.00 last 896.78) - /unlink 17913 (time 1733642275.83 total 20.00 last 894.43) - /unlink 20000 (time 1733642278.09 total 22.26 last 922.89) - /unlink 29041 (time 1733642288.09 total 32.26 last 904.02) - /unlink 38178 (time 1733642298.09 total 42.26 last 913.69) - /unlink 40000 (time 1733642300.30 total 44.47 last 824.26) - /unlink 48703 (time 1733642310.30 total 54.47 last 870.29) - /unlink 57714 (time 1733642320.30 total 64.47 last 901.09) - /unlink 60000 (time 1733642322.72 total 66.89 last 946.90) total: 64768 /unlink in 72.14 seconds: 897.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5441 (time 1733642362.24 total 10.00 last 544.10) - open/close 10000 (time 1733642370.43 total 18.19 last 556.74) - open/close 15290 (time 1733642380.43 total 28.19 last 528.95) - open/close 20000 (time 1733642389.41 total 37.17 last 524.30) - open/close 25048 (time 1733642399.41 total 47.17 last 504.72) - open/close 30000 (time 1733642408.35 total 56.11 last 554.22) - open/close 36060 (time 1733642418.35 total 66.11 last 605.88) - open/close 40000 (time 1733642425.21 total 72.97 last 574.64) - open/close 45853 (time 1733642435.21 total 82.97 last 585.22) - open/close 50000 (time 1733642442.66 total 90.42 last 556.31) - open/close 55756 (time 1733642452.66 total 100.43 last 575.53) - open/close 60000 (time 1733642460.36 total 108.12 last 551.60) total: 64768 open/close in 117.17 seconds: 552.78 ops/second - /unlink 8632 (time 1733642480.31 total 10.00 last 863.19) - /unlink 17156 (time 1733642490.31 total 20.00 last 852.38) - /unlink 20000 (time 1733642493.64 total 23.34 last 852.64) - /unlink 28484 (time 1733642503.64 total 33.34 last 848.36) - /unlink 37108 (time 1733642513.64 total 43.34 last 862.34) - /unlink 40000 (time 1733642516.89 total 46.58 last 891.87) - /unlink 48579 (time 1733642526.89 total 56.58 last 857.81) - /unlink 56762 (time 1733642536.89 total 66.58 last 818.27) - /unlink 60000 (time 1733642540.47 total 70.16 last 904.78) total: 64768 /unlink in 75.38 seconds: 859.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5348 (time 1733642577.89 total 10.00 last 534.74) - open/close 10000 (time 1733642586.21 total 18.32 last 559.31) - open/close 15630 (time 1733642596.21 total 28.32 last 562.87) - open/close 20000 (time 1733642603.94 total 36.05 last 565.43) - open/close 25576 (time 1733642613.94 total 46.05 last 557.52) - open/close 30000 (time 1733642621.73 total 53.84 last 568.22) - open/close 35522 (time 1733642631.73 total 63.84 last 552.16) - open/close 40000 (time 1733642639.86 total 71.97 last 550.50) - open/close 45466 (time 1733642649.86 total 81.97 last 546.51) - open/close 50000 (time 1733642658.50 total 90.61 last 524.87) - open/close 55407 (time 1733642668.50 total 100.61 last 540.70) - open/close 60000 (time 1733642676.91 total 109.02 last 546.44) total: 64768 open/close in 117.96 seconds: 549.05 ops/second - /unlink 8971 (time 1733642696.75 total 10.00 last 897.10) - /unlink 18323 (time 1733642706.75 total 20.00 last 935.18) - /unlink 27675 (time 1733642716.75 total 30.00 last 935.15) - /unlink 30000 (time 1733642719.12 total 32.38 last 977.99) - /unlink 39280 (time 1733642729.12 total 42.38 last 927.92) - /unlink 48491 (time 1733642739.12 total 52.38 last 921.10) - /unlink 57788 (time 1733642749.12 total 62.38 last 929.66) - /unlink 60000 (time 1733642751.48 total 64.73 last 940.21) total: 64768 /unlink in 69.72 seconds: 928.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5594 (time 1733642797.56 total 10.00 last 559.37) - open/close 10000 (time 1733642805.54 total 17.98 last 552.35) - open/close 15376 (time 1733642815.54 total 27.98 last 537.59) - open/close 20000 (time 1733642823.97 total 36.41 last 548.37) - open/close 25565 (time 1733642833.97 total 46.41 last 556.45) - open/close 30000 (time 1733642841.85 total 54.29 last 562.98) - open/close 35495 (time 1733642851.85 total 64.29 last 549.48) - open/close 40000 (time 1733642859.98 total 72.43 last 553.56) - open/close 45756 (time 1733642869.99 total 82.43 last 575.55) - open/close 50000 (time 1733642877.43 total 89.87 last 570.13) - open/close 55650 (time 1733642887.43 total 99.87 last 565.00) - open/close 60000 (time 1733642895.16 total 107.60 last 563.04) total: 64768 open/close in 116.07 seconds: 558.02 ops/second - /unlink 9584 (time 1733642914.44 total 10.00 last 958.38) - /unlink 18968 (time 1733642924.44 total 20.00 last 938.31) - /unlink 28461 (time 1733642934.44 total 30.00 last 949.20) - /unlink 37961 (time 1733642944.44 total 40.00 last 949.97) - /unlink 40000 (time 1733642946.62 total 42.19 last 932.47) - /unlink 49435 (time 1733642956.62 total 52.19 last 943.43) - /unlink 59130 (time 1733642966.63 total 62.19 last 969.40) total: 64768 /unlink in 67.89 seconds: 953.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5838 (time 1733643013.01 total 10.00 last 583.76) - open/close 10000 (time 1733643020.27 total 17.26 last 573.52) - open/close 15260 (time 1733643030.27 total 27.26 last 525.91) - open/close 19626 (time 1733643040.27 total 37.26 last 436.55) - open/close 23214 (time 1733643050.27 total 47.26 last 358.75) - open/close 28924 (time 1733643060.27 total 57.26 last 570.92) - open/close 33335 (time 1733643070.28 total 67.26 last 441.05) - open/close 38598 (time 1733643080.28 total 77.27 last 526.23) - open/close 40000 (time 1733643083.59 total 80.57 last 423.77) - open/close 45369 (time 1733643093.59 total 90.58 last 536.84) - open/close 50000 (time 1733643101.83 total 98.82 last 561.99) - open/close 55692 (time 1733643111.83 total 108.82 last 569.19) - open/close 60000 (time 1733643119.30 total 116.29 last 576.68) total: 64768 open/close in 124.66 seconds: 519.57 ops/second - /unlink 9719 (time 1733643138.52 total 10.00 last 971.84) - /unlink 19459 (time 1733643148.52 total 20.00 last 973.91) - /unlink 29124 (time 1733643158.52 total 30.00 last 966.48) - /unlink 37648 (time 1733643168.52 total 40.00 last 852.33) - /unlink 40000 (time 1733643170.88 total 42.36 last 998.05) - /unlink 49663 (time 1733643180.88 total 52.36 last 966.20) - /unlink 59549 (time 1733643190.88 total 62.36 last 988.53) total: 64768 /unlink in 67.37 seconds: 961.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server unloading modules via unload_modules_local on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 PASS 106 (1136s) Stopping clients: oleg111-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg111-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 oleg111-server: oleg111-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: detected 4 online CPUs by sysfs oleg111-server: Force libcfs to create 2 CPU partitions oleg111-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg111-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1208 sec ========== 02:34:22 (1733643262) === conf-sanity: start cleanup 02:34:22 (1733643262) === === conf-sanity: finish cleanup 02:34:23 (1733643263) ===