-----============= acceptance-small: conf-sanity ============----- Mon Apr 7 05:56:37 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg116-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg116-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg116-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg116-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:58:05 (1744019885) Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg116-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg116-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5533 (time 1744019925.79 total 10.00 last 553.23) - open/close 10000 (time 1744019934.40 total 18.62 last 518.49) - open/close 15534 (time 1744019944.40 total 28.62 last 553.37) - open/close 20000 (time 1744019952.07 total 36.29 last 582.10) - open/close 25701 (time 1744019962.08 total 46.29 last 570.05) - open/close 30000 (time 1744019969.21 total 53.42 last 602.70) - open/close 35946 (time 1744019979.21 total 63.42 last 594.56) - open/close 40000 (time 1744019986.70 total 70.92 last 540.89) - open/close 45658 (time 1744019996.70 total 80.92 last 565.77) - open/close 50000 (time 1744020004.65 total 88.86 last 546.83) - open/close 55371 (time 1744020014.65 total 98.86 last 537.00) - open/close 60000 (time 1744020023.17 total 107.39 last 543.10) total: 64768 open/close in 115.94 seconds: 558.66 ops/second - /unlink 9108 (time 1744020042.58 total 10.00 last 910.71) - /unlink 18857 (time 1744020052.58 total 20.00 last 974.87) - /unlink 28206 (time 1744020062.58 total 30.00 last 934.59) - /unlink 30000 (time 1744020064.86 total 32.29 last 784.80) - /unlink 38922 (time 1744020074.87 total 42.29 last 892.17) - /unlink 47391 (time 1744020084.87 total 52.29 last 846.89) - /unlink 50000 (time 1744020088.43 total 55.86 last 731.46) - /unlink 56609 (time 1744020098.43 total 65.86 last 660.88) - /unlink 60000 (time 1744020102.97 total 70.40 last 747.05) total: 64768 /unlink in 76.66 seconds: 844.85 ops/second Waiting for MDT destroys to complete - open/close 4999 (time 1744020204.46 total 10.00 last 499.87) - open/close 9815 (time 1744020214.46 total 20.00 last 481.58) - open/close 14620 (time 1744020224.46 total 30.00 last 480.43) - open/close 19537 (time 1744020234.46 total 40.00 last 491.62) - open/close 24060 (time 1744020244.47 total 50.01 last 452.24) - open/close 28903 (time 1744020254.47 total 60.01 last 484.22) - open/close 30000 (time 1744020256.66 total 62.20 last 500.71) - open/close 35278 (time 1744020266.66 total 72.20 last 527.77) - open/close 40000 (time 1744020275.81 total 81.35 last 516.20) - open/close 45156 (time 1744020285.81 total 91.35 last 515.60) - open/close 50000 (time 1744020295.31 total 100.85 last 509.48) - open/close 55150 (time 1744020305.32 total 110.86 last 514.92) - open/close 60000 (time 1744020314.65 total 120.19 last 519.34) total: 64768 open/close in 129.15 seconds: 501.51 ops/second - /unlink 9468 (time 1744020334.50 total 10.00 last 946.76) - /unlink 19124 (time 1744020344.50 total 20.00 last 965.53) - /unlink 28775 (time 1744020354.50 total 30.00 last 965.08) - /unlink 38519 (time 1744020364.50 total 40.00 last 974.36) - /unlink 47819 (time 1744020374.50 total 50.00 last 929.98) - /unlink 50000 (time 1744020376.67 total 52.17 last 1004.86) - /unlink 59836 (time 1744020386.67 total 62.17 last 983.52) total: 64768 /unlink in 67.14 seconds: 964.72 ops/second Waiting for MDT destroys to complete - open/close 5386 (time 1744020495.11 total 10.00 last 538.51) - open/close 10000 (time 1744020503.88 total 18.76 last 526.53) - open/close 15381 (time 1744020513.88 total 28.77 last 538.06) - open/close 20000 (time 1744020522.74 total 37.63 last 521.08) - open/close 24523 (time 1744020532.74 total 47.63 last 452.24) - open/close 29020 (time 1744020542.74 total 57.63 last 449.70) - open/close 34198 (time 1744020552.75 total 67.63 last 517.73) - open/close 39431 (time 1744020562.75 total 77.63 last 523.25) - open/close 44631 (time 1744020572.75 total 87.63 last 519.99) - open/close 49892 (time 1744020582.75 total 97.63 last 526.08) - open/close 55052 (time 1744020592.75 total 107.63 last 516.00) - open/close 60000 (time 1744020602.53 total 117.42 last 505.74) total: 64768 open/close in 127.16 seconds: 509.33 ops/second - /unlink 8545 (time 1744020623.16 total 10.00 last 854.45) - /unlink 17591 (time 1744020633.16 total 20.00 last 904.59) - /unlink 20000 (time 1744020635.72 total 22.56 last 940.27) - /unlink 28756 (time 1744020645.72 total 32.56 last 875.57) - /unlink 37843 (time 1744020655.72 total 42.56 last 908.69) - /unlink 40000 (time 1744020658.14 total 44.98 last 890.69) - /unlink 49060 (time 1744020668.14 total 54.99 last 905.94) - /unlink 58321 (time 1744020678.14 total 64.99 last 926.01) total: 64768 /unlink in 71.72 seconds: 903.08 ops/second Waiting for MDT destroys to complete - open/close 4840 (time 1744020785.29 total 10.00 last 483.98) - open/close 9905 (time 1744020795.29 total 20.00 last 506.42) - open/close 14931 (time 1744020805.29 total 30.00 last 502.53) - open/close 20000 (time 1744020815.03 total 39.74 last 520.74) - open/close 23383 (time 1744020825.03 total 49.74 last 338.22) - open/close 28459 (time 1744020835.03 total 59.74 last 507.59) - open/close 30000 (time 1744020839.14 total 63.85 last 375.36) - open/close 32995 (time 1744020849.14 total 73.85 last 299.43) - open/close 37243 (time 1744020859.14 total 83.85 last 424.69) - open/close 40000 (time 1744020868.25 total 92.96 last 302.62) - open/close 43198 (time 1744020878.25 total 102.96 last 319.76) - open/close 46895 (time 1744020888.25 total 112.96 last 369.64) - open/close 50000 (time 1744020896.22 total 120.93 last 389.84) - open/close 54128 (time 1744020906.22 total 130.93 last 412.74) - open/close 58069 (time 1744020916.22 total 140.93 last 394.04) - open/close 60000 (time 1744020920.37 total 145.08 last 465.57) - open/close 63850 (time 1744020930.37 total 155.08 last 384.91) total: 64768 open/close in 158.07 seconds: 409.73 ops/second - /unlink 8465 (time 1744020944.36 total 10.00 last 846.44) - /unlink 10000 (time 1744020946.51 total 12.15 last 713.17) - /unlink 16089 (time 1744020956.51 total 22.15 last 608.80) - /unlink 20000 (time 1744020964.36 total 30.00 last 498.50) - /unlink 26981 (time 1744020974.36 total 40.00 last 698.06) - /unlink 30000 (time 1744020979.00 total 44.64 last 651.17) - /unlink 36263 (time 1744020989.00 total 54.64 last 626.25) - /unlink 40000 (time 1744020994.59 total 60.23 last 667.79) - /unlink 47292 (time 1744021004.59 total 70.23 last 729.17) - /unlink 50000 (time 1744021007.71 total 73.35 last 868.28) - /unlink 55908 (time 1744021017.71 total 83.36 last 590.66) - /unlink 60000 (time 1744021022.90 total 88.54 last 789.72) total: 64768 /unlink in 96.08 seconds: 674.08 ops/second Waiting for MDT destroys to complete - open/close 3301 (time 1744021105.80 total 10.00 last 330.05) - open/close 8006 (time 1744021115.80 total 20.00 last 470.44) - open/close 10000 (time 1744021119.73 total 23.93 last 507.56) - open/close 15331 (time 1744021129.73 total 33.93 last 533.08) - open/close 19991 (time 1744021139.73 total 43.93 last 465.99) - open/close 25177 (time 1744021149.73 total 53.93 last 518.54) - open/close 30000 (time 1744021159.29 total 63.49 last 504.59) - open/close 35356 (time 1744021169.29 total 73.49 last 535.56) - open/close 40000 (time 1744021178.21 total 82.41 last 520.94) - open/close 45415 (time 1744021188.21 total 92.41 last 541.40) - open/close 50000 (time 1744021196.69 total 100.89 last 540.39) - open/close 55334 (time 1744021206.69 total 110.89 last 533.38) - open/close 60000 (time 1744021215.56 total 119.76 last 526.01) - open/close 64563 (time 1744021225.57 total 129.77 last 456.21) total: 64768 open/close in 130.21 seconds: 497.41 ops/second - /unlink 7694 (time 1744021237.09 total 10.00 last 769.37) - /unlink 10000 (time 1744021239.71 total 12.62 last 880.28) - /unlink 18905 (time 1744021249.71 total 22.62 last 890.44) - /unlink 28291 (time 1744021259.71 total 32.62 last 938.55) - /unlink 34744 (time 1744021269.71 total 42.62 last 645.22) - /unlink 40000 (time 1744021276.67 total 49.58 last 755.37) - /unlink 49845 (time 1744021286.67 total 59.58 last 984.49) - /unlink 58601 (time 1744021296.67 total 69.58 last 875.54) total: 64768 /unlink in 75.74 seconds: 855.13 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg116-server unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 PASS 106 (1531s) Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1637 sec ========== 06:23:54 (1744021434) === conf-sanity: start cleanup 06:23:54 (1744021434) === === conf-sanity: finish cleanup 06:23:55 (1744021435) ===