-----============= acceptance-small: conf-sanity ============----- Mon Mar 31 16:51:25 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg334-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg334-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg334-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg334-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg334-server oleg334-server: oleg334-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg334-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg334-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg334-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:52:50 (1743454370) Stopping clients: oleg334-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg334-client.virtnet /mnt/lustre2 (opts:-f) oleg334-server: oleg334-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg334-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg334-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg334-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5271 (time 1743454410.02 total 10.00 last 527.03) - open/close 10000 (time 1743454418.96 total 18.95 last 528.67) - open/close 15243 (time 1743454428.96 total 28.95 last 524.27) - open/close 20000 (time 1743454437.81 total 37.79 last 537.70) - open/close 25366 (time 1743454447.81 total 47.80 last 536.53) - open/close 30000 (time 1743454456.61 total 56.59 last 526.85) - open/close 35297 (time 1743454466.61 total 66.59 last 529.62) - open/close 40000 (time 1743454475.51 total 75.50 last 528.24) - open/close 45226 (time 1743454485.51 total 85.50 last 522.52) - open/close 50000 (time 1743454494.69 total 94.67 last 520.33) - open/close 55235 (time 1743454504.69 total 104.67 last 523.46) - open/close 60000 (time 1743454513.80 total 113.79 last 522.91) total: 64768 open/close in 122.96 seconds: 526.72 ops/second - /unlink 9427 (time 1743454533.84 total 10.00 last 942.69) - /unlink 18904 (time 1743454543.84 total 20.00 last 947.68) - /unlink 28492 (time 1743454553.84 total 30.00 last 958.76) - /unlink 38123 (time 1743454563.84 total 40.00 last 963.04) - /unlink 47859 (time 1743454573.84 total 50.00 last 973.56) - /unlink 50000 (time 1743454576.09 total 52.25 last 952.42) - /unlink 59701 (time 1743454586.09 total 62.25 last 970.07) total: 64768 /unlink in 67.39 seconds: 961.09 ops/second Waiting for MDT destroys to complete - open/close 3945 (time 1743454693.02 total 10.00 last 394.43) - open/close 9021 (time 1743454703.02 total 20.00 last 507.56) - open/close 12767 (time 1743454713.02 total 30.01 last 374.48) - open/close 16963 (time 1743454723.02 total 40.01 last 419.52) - open/close 19657 (time 1743454733.02 total 50.01 last 269.34) - open/close 22777 (time 1743454743.03 total 60.01 last 311.89) - open/close 25604 (time 1743454753.03 total 70.02 last 282.64) - open/close 29051 (time 1743454763.03 total 80.02 last 344.69) - open/close 30000 (time 1743454765.20 total 82.19 last 436.58) - open/close 33710 (time 1743454775.21 total 92.19 last 370.92) - open/close 37644 (time 1743454785.21 total 102.19 last 393.33) - open/close 40000 (time 1743454789.92 total 106.90 last 500.39) - open/close 45022 (time 1743454799.92 total 116.90 last 502.14) - open/close 49098 (time 1743454809.92 total 126.90 last 407.54) - open/close 50000 (time 1743454812.90 total 129.89 last 302.62) - open/close 54095 (time 1743454822.90 total 139.89 last 409.44) - open/close 57402 (time 1743454832.92 total 149.91 last 329.95) - open/close 60000 (time 1743454839.31 total 156.30 last 406.56) total: 64768 open/close in 165.61 seconds: 391.10 ops/second - /unlink 8048 (time 1743454859.46 total 10.00 last 804.70) - /unlink 10000 (time 1743454862.57 total 13.11 last 627.86) - /unlink 17967 (time 1743454872.57 total 23.11 last 796.69) - /unlink 20000 (time 1743454875.18 total 25.72 last 779.41) - /unlink 27789 (time 1743454885.18 total 35.72 last 778.85) - /unlink 30000 (time 1743454888.10 total 38.64 last 757.13) - /unlink 35903 (time 1743454898.10 total 48.64 last 590.26) - /unlink 40000 (time 1743454903.51 total 54.04 last 758.19) - /unlink 46505 (time 1743454913.51 total 64.04 last 650.45) - /unlink 50000 (time 1743454917.76 total 68.30 last 821.27) - /unlink 56385 (time 1743454927.76 total 78.30 last 638.49) - /unlink 60000 (time 1743454933.06 total 83.60 last 682.14) total: 64768 /unlink in 90.08 seconds: 719.01 ops/second Waiting for MDT destroys to complete - open/close 5087 (time 1743455019.15 total 10.00 last 508.60) - open/close 8764 (time 1743455029.16 total 20.01 last 367.57) - open/close 10000 (time 1743455032.05 total 22.90 last 427.69) - open/close 12833 (time 1743455042.05 total 32.90 last 283.24) - open/close 15591 (time 1743455052.05 total 42.90 last 275.75) - open/close 18451 (time 1743455062.05 total 52.90 last 285.91) - open/close 20000 (time 1743455066.21 total 57.06 last 372.61) - open/close 24086 (time 1743455076.21 total 67.06 last 408.53) - open/close 29483 (time 1743455086.21 total 77.06 last 539.66) - open/close 34744 (time 1743455096.22 total 87.06 last 526.04) - open/close 39375 (time 1743455106.22 total 97.06 last 463.04) - open/close 43775 (time 1743455116.22 total 107.07 last 439.96) - open/close 48000 (time 1743455126.22 total 117.07 last 422.49) - open/close 50000 (time 1743455132.72 total 123.57 last 307.58) - open/close 52639 (time 1743455142.72 total 133.57 last 263.89) - open/close 55562 (time 1743455152.72 total 143.57 last 292.23) - open/close 58949 (time 1743455162.73 total 153.57 last 338.60) - open/close 60000 (time 1743455165.36 total 156.20 last 399.66) - open/close 64544 (time 1743455175.36 total 166.21 last 454.33) total: 64768 open/close in 166.82 seconds: 388.24 ops/second - /unlink 6859 (time 1743455187.00 total 10.00 last 685.85) - /unlink 10000 (time 1743455191.18 total 14.18 last 750.73) - /unlink 17717 (time 1743455201.19 total 24.19 last 771.63) - /unlink 20000 (time 1743455204.23 total 27.23 last 749.42) - /unlink 28253 (time 1743455214.23 total 37.23 last 825.27) - /unlink 30000 (time 1743455216.88 total 39.88 last 659.18) - /unlink 36710 (time 1743455226.88 total 49.88 last 670.94) - /unlink 40000 (time 1743455231.45 total 54.45 last 719.84) - /unlink 45980 (time 1743455241.45 total 64.45 last 597.95) - /unlink 50000 (time 1743455246.64 total 69.64 last 775.00) - /unlink 57738 (time 1743455256.64 total 79.64 last 773.76) - /unlink 60000 (time 1743455259.40 total 82.40 last 820.33) total: 64768 /unlink in 88.07 seconds: 735.42 ops/second Waiting for MDT destroys to complete - open/close 3158 (time 1743455339.41 total 10.00 last 315.75) - open/close 6210 (time 1743455349.42 total 20.00 last 305.13) - open/close 9205 (time 1743455359.42 total 30.00 last 299.47) - open/close 10000 (time 1743455362.12 total 32.71 last 293.71) - open/close 13163 (time 1743455372.12 total 42.71 last 316.30) - open/close 16437 (time 1743455382.13 total 52.71 last 327.31) - open/close 19708 (time 1743455392.13 total 62.72 last 327.00) - open/close 23045 (time 1743455402.13 total 72.72 last 333.65) - open/close 26383 (time 1743455412.13 total 82.72 last 333.68) - open/close 29281 (time 1743455422.14 total 92.73 last 289.71) - open/close 30000 (time 1743455424.88 total 95.47 last 262.05) - open/close 32792 (time 1743455434.88 total 105.47 last 279.16) - open/close 36918 (time 1743455444.88 total 115.47 last 412.52) - open/close 40000 (time 1743455451.29 total 121.88 last 481.19) - open/close 44934 (time 1743455461.29 total 131.88 last 493.29) - open/close 48813 (time 1743455471.29 total 141.88 last 387.80) - open/close 50000 (time 1743455474.70 total 145.29 last 348.12) - open/close 53444 (time 1743455484.71 total 155.29 last 344.35) - open/close 57098 (time 1743455494.71 total 165.30 last 365.29) - open/close 60000 (time 1743455502.79 total 173.38 last 359.18) - open/close 64332 (time 1743455512.79 total 183.38 last 432.97) total: 64768 open/close in 184.99 seconds: 350.12 ops/second - /unlink 4962 (time 1743455525.35 total 10.00 last 496.16) - /unlink 10000 (time 1743455531.18 total 15.83 last 864.09) - /unlink 18547 (time 1743455541.18 total 25.83 last 854.64) - /unlink 27670 (time 1743455551.18 total 35.83 last 912.29) - /unlink 30000 (time 1743455554.85 total 39.50 last 634.36) - /unlink 38534 (time 1743455564.85 total 49.51 last 853.38) - /unlink 48877 (time 1743455574.85 total 59.51 last 1034.24) - /unlink 58573 (time 1743455584.85 total 69.51 last 969.55) total: 64768 /unlink in 76.20 seconds: 849.98 ops/second Waiting for MDT destroys to complete - open/close 3495 (time 1743455685.54 total 10.00 last 349.46) - open/close 7540 (time 1743455695.55 total 20.00 last 404.42) - open/close 10000 (time 1743455701.26 total 25.72 last 430.25) - open/close 14632 (time 1743455711.26 total 35.72 last 463.19) - open/close 19243 (time 1743455721.27 total 45.72 last 461.06) - open/close 24029 (time 1743455731.27 total 55.72 last 478.56) - open/close 29541 (time 1743455741.27 total 65.72 last 551.13) - open/close 35446 (time 1743455751.27 total 75.72 last 590.44) - open/close 40000 (time 1743455759.20 total 83.66 last 574.25) - open/close 45491 (time 1743455769.20 total 93.66 last 549.02) - open/close 50000 (time 1743455778.19 total 102.65 last 501.59) - open/close 54906 (time 1743455788.19 total 112.65 last 490.52) - open/close 60000 (time 1743455798.18 total 122.64 last 509.85) total: 64768 open/close in 132.35 seconds: 489.36 ops/second - /unlink 10000 (time 1743455818.47 total 9.74 last 1026.86) - /unlink 20000 (time 1743455827.78 total 19.05 last 1073.66) - /unlink 29470 (time 1743455837.79 total 29.05 last 946.96) - /unlink 38764 (time 1743455847.79 total 39.05 last 929.34) - /unlink 48860 (time 1743455857.79 total 49.05 last 1009.53) - /unlink 57779 (time 1743455867.79 total 59.05 last 891.86) - /unlink 60000 (time 1743455870.19 total 61.46 last 922.53) total: 64768 /unlink in 66.49 seconds: 974.11 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 PASS 106 (1606s) Stopping clients: oleg334-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg334-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 oleg334-server: oleg334-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1709 sec ========== 17:19:54 (1743455994) === conf-sanity: start cleanup 17:19:55 (1743455995) === === conf-sanity: finish cleanup 17:19:55 (1743455995) ===