-----============= acceptance-small: conf-sanity ============----- Tue Mar 25 07:39:29 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg335-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg335-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg335-server oleg335-server: oleg335-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg335-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg335-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg335-server: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg335-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:40:29 (1742902829) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:-f) oleg335-server: oleg335-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg335-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg335-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3361 (time 1742902868.30 total 10.00 last 336.08) - open/close 6359 (time 1742902878.30 total 20.00 last 299.77) - open/close 8182 (time 1742902888.30 total 30.00 last 182.30) - open/close 10000 (time 1742902897.92 total 39.63 last 188.92) - open/close 12283 (time 1742902907.93 total 49.63 last 228.23) - open/close 15027 (time 1742902917.93 total 59.63 last 274.34) - open/close 17763 (time 1742902927.93 total 69.63 last 273.59) - open/close 20000 (time 1742902935.57 total 77.27 last 292.77) - open/close 23037 (time 1742902945.57 total 87.27 last 303.65) - open/close 26552 (time 1742902955.57 total 97.27 last 351.48) - open/close 29393 (time 1742902965.58 total 107.28 last 283.99) - open/close 30000 (time 1742902968.04 total 109.74 last 246.80) - open/close 32702 (time 1742902978.04 total 119.74 last 270.13) - open/close 34614 (time 1742902988.04 total 129.74 last 191.16) - open/close 36446 (time 1742902998.04 total 139.75 last 183.14) - open/close 38711 (time 1742903008.04 total 149.75 last 226.48) - open/close 40000 (time 1742903013.72 total 155.42 last 227.23) - open/close 42649 (time 1742903023.72 total 165.42 last 264.81) - open/close 45064 (time 1742903033.72 total 175.43 last 241.40) - open/close 47362 (time 1742903043.73 total 185.43 last 229.76) - open/close 49103 (time 1742903053.73 total 195.44 last 173.97) - open/close 50000 (time 1742903058.40 total 200.10 last 192.38) - open/close 52722 (time 1742903068.40 total 210.10 last 272.11) - open/close 55344 (time 1742903078.40 total 220.11 last 262.10) - open/close 57448 (time 1742903088.41 total 230.11 last 210.38) - open/close 59425 (time 1742903098.41 total 240.11 last 197.66) - open/close 62016 (time 1742903108.41 total 250.11 last 259.09) - open/close 63890 (time 1742903118.41 total 260.11 last 187.40) total: 64768 open/close in 265.34 seconds: 244.09 ops/second - /unlink 3067 (time 1742903134.98 total 10.00 last 306.63) - /unlink 6335 (time 1742903144.98 total 20.00 last 326.76) - /unlink 10000 (time 1742903154.38 total 29.40 last 389.98) - /unlink 14301 (time 1742903164.38 total 39.40 last 430.07) - /unlink 18115 (time 1742903174.38 total 49.40 last 381.35) - /unlink 20000 (time 1742903179.35 total 54.38 last 379.06) - /unlink 24403 (time 1742903189.35 total 64.38 last 440.21) - /unlink 27923 (time 1742903199.36 total 74.38 last 351.89) - /unlink 30000 (time 1742903204.81 total 79.83 last 381.02) - /unlink 34474 (time 1742903214.81 total 89.83 last 447.33) - /unlink 38099 (time 1742903224.81 total 99.84 last 362.34) - /unlink 40000 (time 1742903230.26 total 105.28 last 349.23) - /unlink 43585 (time 1742903240.26 total 115.28 last 358.50) - /unlink 46694 (time 1742903250.26 total 125.28 last 310.88) - /unlink 50000 (time 1742903259.02 total 134.05 last 377.10) - /unlink 53359 (time 1742903269.02 total 144.05 last 335.87) - /unlink 57661 (time 1742903279.03 total 154.05 last 430.10) - /unlink 60000 (time 1742903283.05 total 158.08 last 581.12) total: 64768 /unlink in 167.16 seconds: 387.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3137 (time 1742903317.57 total 10.00 last 313.59) - open/close 6073 (time 1742903327.57 total 20.00 last 293.56) - open/close 9793 (time 1742903337.57 total 30.01 last 371.95) - open/close 13608 (time 1742903347.57 total 40.01 last 381.45) - open/close 18105 (time 1742903357.57 total 50.01 last 449.60) - open/close 20000 (time 1742903361.75 total 54.19 last 453.16) - open/close 24045 (time 1742903371.76 total 64.19 last 404.44) - open/close 28093 (time 1742903381.76 total 74.19 last 404.75) - open/close 30000 (time 1742903386.40 total 78.83 last 410.89) - open/close 34899 (time 1742903396.40 total 88.83 last 489.90) - open/close 40000 (time 1742903406.03 total 98.47 last 529.46) - open/close 45115 (time 1742903416.03 total 108.47 last 511.45) - open/close 49569 (time 1742903426.03 total 118.47 last 445.39) - open/close 53567 (time 1742903436.04 total 128.47 last 399.76) - open/close 57079 (time 1742903446.04 total 138.47 last 351.14) - open/close 60000 (time 1742903454.45 total 146.88 last 347.36) - open/close 63745 (time 1742903464.45 total 156.88 last 374.41) total: 64768 open/close in 160.02 seconds: 404.74 ops/second - /unlink 7744 (time 1742903478.43 total 10.00 last 774.32) - /unlink 10000 (time 1742903481.69 total 13.26 last 691.64) - /unlink 16974 (time 1742903491.69 total 23.26 last 697.36) - /unlink 20000 (time 1742903495.49 total 27.06 last 796.35) - /unlink 26738 (time 1742903505.49 total 37.06 last 673.71) - /unlink 30000 (time 1742903510.70 total 42.27 last 627.03) - /unlink 36246 (time 1742903520.70 total 52.27 last 624.55) - /unlink 40000 (time 1742903526.36 total 57.94 last 662.25) - /unlink 46258 (time 1742903536.36 total 67.94 last 625.79) - /unlink 50000 (time 1742903542.23 total 73.80 last 637.91) - /unlink 56904 (time 1742903552.23 total 83.80 last 690.31) - /unlink 60000 (time 1742903557.04 total 88.61 last 644.13) total: 64768 /unlink in 95.85 seconds: 675.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4202 (time 1742903592.57 total 10.00 last 420.12) - open/close 8379 (time 1742903602.57 total 20.00 last 417.62) - open/close 10000 (time 1742903606.71 total 24.15 last 391.30) - open/close 14674 (time 1742903616.72 total 34.15 last 467.36) - open/close 19564 (time 1742903626.72 total 44.15 last 488.97) - open/close 24144 (time 1742903636.72 total 54.15 last 457.99) - open/close 29865 (time 1742903646.72 total 64.15 last 572.04) - open/close 35396 (time 1742903656.72 total 74.15 last 553.08) - open/close 40000 (time 1742903664.98 total 82.41 last 557.01) - open/close 44807 (time 1742903674.99 total 92.42 last 480.53) - open/close 49948 (time 1742903684.99 total 102.42 last 514.07) - open/close 55248 (time 1742903694.99 total 112.42 last 529.96) - open/close 60000 (time 1742903704.75 total 122.19 last 486.61) - open/close 64630 (time 1742903714.78 total 132.22 last 461.59) total: 64768 open/close in 132.50 seconds: 488.81 ops/second - /unlink 9895 (time 1742903725.93 total 10.00 last 989.42) - /unlink 19528 (time 1742903735.93 total 20.00 last 963.25) - /unlink 28368 (time 1742903745.93 total 30.00 last 883.93) - /unlink 37761 (time 1742903755.93 total 40.00 last 939.22) - /unlink 40000 (time 1742903758.11 total 42.18 last 1027.34) - /unlink 50000 (time 1742903767.53 total 51.61 last 1060.91) - /unlink 58131 (time 1742903777.53 total 61.61 last 813.06) - /unlink 60000 (time 1742903780.05 total 64.12 last 742.83) total: 64768 /unlink in 69.00 seconds: 938.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5170 (time 1742903827.78 total 10.00 last 516.99) - open/close 9802 (time 1742903837.78 total 20.00 last 463.17) - open/close 15053 (time 1742903847.78 total 30.00 last 525.04) - open/close 20000 (time 1742903857.64 total 39.86 last 501.89) - open/close 26068 (time 1742903867.64 total 49.86 last 606.72) - open/close 30000 (time 1742903875.56 total 57.77 last 496.85) - open/close 35567 (time 1742903885.56 total 67.78 last 556.63) - open/close 40000 (time 1742903892.89 total 75.11 last 604.25) - open/close 45389 (time 1742903902.89 total 85.11 last 538.86) - open/close 50000 (time 1742903912.20 total 94.42 last 495.49) - open/close 55165 (time 1742903922.20 total 104.42 last 516.46) - open/close 60000 (time 1742903931.67 total 113.89 last 510.48) total: 64768 open/close in 123.34 seconds: 525.12 ops/second - /unlink 9190 (time 1742903951.96 total 10.00 last 918.97) - /unlink 18329 (time 1742903961.96 total 20.00 last 913.84) - /unlink 27650 (time 1742903971.96 total 30.00 last 932.02) - /unlink 30000 (time 1742903974.41 total 32.46 last 957.72) - /unlink 38433 (time 1742903984.41 total 42.46 last 843.21) - /unlink 46318 (time 1742903994.42 total 52.46 last 788.41) - /unlink 50000 (time 1742903998.92 total 56.96 last 818.11) - /unlink 58685 (time 1742904008.92 total 66.96 last 868.49) total: 64768 /unlink in 73.47 seconds: 881.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4940 (time 1742904053.21 total 10.00 last 493.94) - open/close 10000 (time 1742904063.00 total 19.79 last 516.85) - open/close 15004 (time 1742904073.00 total 29.79 last 500.33) - open/close 19811 (time 1742904083.00 total 39.79 last 480.66) - open/close 24460 (time 1742904093.01 total 49.80 last 464.81) - open/close 29283 (time 1742904103.01 total 59.80 last 482.20) - open/close 34570 (time 1742904113.01 total 69.80 last 528.64) - open/close 39909 (time 1742904123.01 total 79.80 last 533.90) - open/close 45175 (time 1742904133.01 total 89.80 last 526.57) - open/close 50000 (time 1742904142.42 total 99.21 last 512.64) - open/close 54826 (time 1742904152.42 total 109.21 last 482.57) - open/close 59974 (time 1742904162.42 total 119.21 last 514.78) total: 64768 open/close in 129.07 seconds: 501.80 ops/second - /unlink 8784 (time 1742904183.14 total 10.00 last 878.34) - /unlink 17687 (time 1742904193.14 total 20.00 last 890.26) - /unlink 20000 (time 1742904195.81 total 22.67 last 868.06) - /unlink 28264 (time 1742904205.81 total 32.67 last 826.21) - /unlink 30000 (time 1742904208.02 total 34.88 last 784.91) - /unlink 38546 (time 1742904218.02 total 44.88 last 854.54) - /unlink 47210 (time 1742904228.03 total 54.88 last 866.31) - /unlink 50000 (time 1742904231.05 total 57.91 last 922.46) - /unlink 59293 (time 1742904241.05 total 67.91 last 929.29) total: 64768 /unlink in 73.79 seconds: 877.72 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 PASS 106 (1464s) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 oleg335-server: oleg335-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions oleg335-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg335-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1541 sec ========== 08:05:10 (1742904310) === conf-sanity: start cleanup 08:05:10 (1742904310) === === conf-sanity: finish cleanup 08:05:11 (1742904311) ===