-----============= acceptance-small: conf-sanity ============----- Tue Aug 19 16:56:22 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg423-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg423-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg423-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg423-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg423-server oleg423-server: oleg423-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg423-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg423-server: oleg423-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg423-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:57:28 (1755637048) Stopping clients: oleg423-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg423-client.virtnet /mnt/lustre2 (opts:-f) oleg423-server: oleg423-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg423-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4631 (time 1755637094.12 total 10.00 last 463.05) - open/close 9304 (time 1755637104.12 total 20.00 last 467.29) - open/close 13992 (time 1755637114.12 total 30.00 last 468.73) - open/close 18515 (time 1755637124.12 total 40.00 last 452.28) - open/close 20000 (time 1755637127.33 total 43.21 last 463.04) - open/close 24636 (time 1755637137.33 total 53.21 last 463.58) - open/close 29263 (time 1755637147.33 total 63.21 last 462.64) - open/close 34031 (time 1755637157.33 total 73.21 last 476.78) - open/close 38672 (time 1755637167.33 total 83.21 last 464.01) - open/close 40000 (time 1755637170.32 total 86.20 last 444.45) - open/close 44539 (time 1755637180.32 total 96.20 last 453.83) - open/close 48865 (time 1755637190.32 total 106.21 last 432.54) - open/close 50000 (time 1755637193.02 total 108.90 last 421.20) - open/close 54466 (time 1755637203.02 total 118.90 last 446.53) - open/close 58932 (time 1755637213.02 total 128.90 last 446.57) - open/close 60000 (time 1755637215.41 total 131.29 last 446.71) - open/close 64436 (time 1755637225.41 total 141.29 last 443.53) total: 64768 open/close in 142.10 seconds: 455.78 ops/second - /unlink 7311 (time 1755637237.68 total 10.00 last 730.74) - /unlink 10000 (time 1755637240.94 total 13.27 last 824.25) - /unlink 17409 (time 1755637250.94 total 23.27 last 740.88) - /unlink 20000 (time 1755637254.67 total 27.00 last 694.10) - /unlink 27554 (time 1755637264.68 total 37.00 last 755.33) - /unlink 30000 (time 1755637267.63 total 39.96 last 827.38) - /unlink 37283 (time 1755637277.63 total 49.96 last 728.23) - /unlink 40000 (time 1755637280.91 total 53.24 last 829.24) - /unlink 47196 (time 1755637290.91 total 63.24 last 719.56) - /unlink 50000 (time 1755637294.92 total 67.25 last 699.13) - /unlink 57757 (time 1755637304.92 total 77.25 last 775.66) - /unlink 60000 (time 1755637307.14 total 79.47 last 1009.70) total: 64768 /unlink in 85.95 seconds: 753.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4704 (time 1755637344.40 total 10.00 last 470.31) - open/close 9356 (time 1755637354.40 total 20.01 last 465.01) - open/close 14014 (time 1755637364.41 total 30.01 last 465.74) - open/close 18559 (time 1755637374.41 total 40.01 last 454.48) - open/close 20000 (time 1755637377.47 total 43.07 last 470.43) - open/close 24730 (time 1755637387.47 total 53.07 last 472.93) - open/close 29371 (time 1755637397.47 total 63.07 last 464.10) - open/close 33985 (time 1755637407.47 total 73.07 last 461.33) - open/close 38629 (time 1755637417.47 total 83.07 last 464.38) - open/close 40000 (time 1755637420.31 total 85.92 last 482.50) - open/close 44481 (time 1755637430.32 total 95.92 last 448.08) - open/close 49082 (time 1755637440.32 total 105.92 last 460.09) - open/close 50000 (time 1755637442.38 total 107.98 last 445.52) - open/close 54480 (time 1755637452.38 total 117.98 last 447.96) - open/close 58791 (time 1755637462.38 total 127.98 last 431.02) - open/close 60000 (time 1755637465.09 total 130.69 last 446.29) - open/close 64378 (time 1755637475.09 total 140.69 last 437.72) total: 64768 open/close in 141.68 seconds: 457.16 ops/second - /unlink 7430 (time 1755637487.48 total 10.00 last 742.96) - /unlink 10000 (time 1755637490.60 total 13.12 last 823.01) - /unlink 17493 (time 1755637500.60 total 23.12 last 749.27) - /unlink 20000 (time 1755637504.19 total 26.71 last 698.52) - /unlink 27390 (time 1755637514.20 total 36.72 last 738.33) - /unlink 30000 (time 1755637517.49 total 40.02 last 792.18) - /unlink 37947 (time 1755637527.49 total 50.02 last 794.66) - /unlink 40000 (time 1755637530.37 total 52.90 last 712.76) - /unlink 48211 (time 1755637540.37 total 62.90 last 820.96) - /unlink 50000 (time 1755637542.60 total 65.13 last 803.56) - /unlink 58166 (time 1755637552.60 total 75.13 last 816.52) - /unlink 60000 (time 1755637554.67 total 77.19 last 888.96) total: 64768 /unlink in 83.14 seconds: 779.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4520 (time 1755637643.24 total 10.00 last 451.97) - open/close 9266 (time 1755637653.24 total 20.00 last 474.59) - open/close 13627 (time 1755637663.24 total 30.00 last 436.07) - open/close 18407 (time 1755637673.25 total 40.00 last 477.91) - open/close 20000 (time 1755637676.54 total 43.29 last 484.03) - open/close 24725 (time 1755637686.54 total 53.30 last 472.47) - open/close 29448 (time 1755637696.54 total 63.30 last 472.24) - open/close 34069 (time 1755637706.54 total 73.30 last 462.05) - open/close 38599 (time 1755637716.54 total 83.30 last 453.00) - open/close 40000 (time 1755637719.61 total 86.37 last 456.42) - open/close 44683 (time 1755637729.61 total 96.37 last 468.24) - open/close 49335 (time 1755637739.61 total 106.37 last 465.15) - open/close 54071 (time 1755637749.61 total 116.37 last 473.57) - open/close 58541 (time 1755637759.61 total 126.37 last 446.98) - open/close 60000 (time 1755637763.02 total 129.78 last 427.71) - open/close 64409 (time 1755637773.03 total 139.78 last 440.87) total: 64768 open/close in 140.64 seconds: 460.53 ops/second - /unlink 7521 (time 1755637785.20 total 10.00 last 752.06) - /unlink 10000 (time 1755637788.18 total 12.98 last 831.26) - /unlink 17417 (time 1755637798.18 total 22.98 last 741.70) - /unlink 20000 (time 1755637801.74 total 26.55 last 724.74) - /unlink 27671 (time 1755637811.74 total 36.55 last 767.03) - /unlink 30000 (time 1755637814.99 total 39.79 last 717.27) - /unlink 37581 (time 1755637824.99 total 49.79 last 758.08) - /unlink 40000 (time 1755637828.13 total 52.94 last 769.68) - /unlink 47464 (time 1755637838.14 total 62.94 last 746.36) - /unlink 50000 (time 1755637841.67 total 66.47 last 717.26) - /unlink 57546 (time 1755637851.67 total 76.48 last 754.52) - /unlink 60000 (time 1755637854.56 total 79.36 last 849.39) total: 64768 /unlink in 85.75 seconds: 755.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4384 (time 1755637892.17 total 10.00 last 438.36) - open/close 8543 (time 1755637902.17 total 20.00 last 415.89) - open/close 10000 (time 1755637905.21 total 23.04 last 479.15) - open/close 14601 (time 1755637915.21 total 33.04 last 460.07) - open/close 19308 (time 1755637925.21 total 43.04 last 470.64) - open/close 23465 (time 1755637935.21 total 53.04 last 415.69) - open/close 27516 (time 1755637945.21 total 63.05 last 405.02) - open/close 30000 (time 1755637950.51 total 68.35 last 468.71) - open/close 34752 (time 1755637960.51 total 78.35 last 475.11) - open/close 39178 (time 1755637970.51 total 88.35 last 442.52) - open/close 43669 (time 1755637980.52 total 98.35 last 449.06) - open/close 47813 (time 1755637990.52 total 108.35 last 414.37) - open/close 50000 (time 1755637995.84 total 113.67 last 411.19) - open/close 54445 (time 1755638005.84 total 123.67 last 444.49) - open/close 58463 (time 1755638015.84 total 133.67 last 401.69) - open/close 60000 (time 1755638019.37 total 137.20 last 435.39) - open/close 63984 (time 1755638029.37 total 147.20 last 398.32) total: 64768 open/close in 149.21 seconds: 434.06 ops/second - /unlink 6384 (time 1755638042.78 total 10.00 last 638.40) - /unlink 10000 (time 1755638047.74 total 14.96 last 729.77) - /unlink 16406 (time 1755638057.74 total 24.96 last 640.47) - /unlink 20000 (time 1755638062.56 total 29.78 last 745.80) - /unlink 27498 (time 1755638072.57 total 39.78 last 749.18) - /unlink 30000 (time 1755638076.02 total 43.24 last 723.64) - /unlink 37326 (time 1755638086.02 total 53.24 last 732.56) - /unlink 40000 (time 1755638089.77 total 56.99 last 714.32) - /unlink 47341 (time 1755638099.77 total 66.99 last 734.10) - /unlink 50000 (time 1755638102.77 total 69.99 last 884.23) - /unlink 57806 (time 1755638112.78 total 79.99 last 780.54) - /unlink 60000 (time 1755638115.88 total 83.10 last 705.99) total: 64768 /unlink in 89.27 seconds: 725.56 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4656 (time 1755638156.33 total 10.00 last 465.55) - open/close 9390 (time 1755638166.33 total 20.00 last 473.33) - open/close 14120 (time 1755638176.33 total 30.00 last 472.95) - open/close 18756 (time 1755638186.33 total 40.00 last 463.55) - open/close 20000 (time 1755638189.10 total 42.78 last 449.02) - open/close 24339 (time 1755638199.10 total 52.78 last 433.79) - open/close 28689 (time 1755638209.11 total 62.78 last 434.93) - open/close 30000 (time 1755638211.98 total 65.65 last 456.70) - open/close 34778 (time 1755638221.98 total 75.65 last 477.79) - open/close 39280 (time 1755638231.98 total 85.65 last 450.18) - open/close 43670 (time 1755638241.98 total 95.65 last 438.94) - open/close 47731 (time 1755638251.98 total 105.65 last 406.08) - open/close 50000 (time 1755638257.88 total 111.55 last 384.69) - open/close 54267 (time 1755638267.88 total 121.55 last 426.68) - open/close 58663 (time 1755638277.88 total 131.55 last 439.60) - open/close 60000 (time 1755638280.98 total 134.65 last 430.80) - open/close 64522 (time 1755638290.98 total 144.65 last 452.19) total: 64768 open/close in 145.17 seconds: 446.16 ops/second - /unlink 7841 (time 1755638302.87 total 10.00 last 784.03) - /unlink 10000 (time 1755638305.53 total 12.66 last 813.01) - /unlink 17725 (time 1755638315.53 total 22.66 last 772.41) - /unlink 20000 (time 1755638318.80 total 25.93 last 695.56) - /unlink 27754 (time 1755638328.80 total 35.93 last 775.33) - /unlink 30000 (time 1755638331.33 total 38.45 last 889.86) - /unlink 37588 (time 1755638341.33 total 48.45 last 758.76) - /unlink 40000 (time 1755638345.65 total 52.78 last 557.49) - /unlink 47264 (time 1755638355.65 total 62.78 last 726.38) - /unlink 50000 (time 1755638359.43 total 66.56 last 723.66) - /unlink 57888 (time 1755638369.43 total 76.56 last 788.80) - /unlink 60000 (time 1755638371.78 total 78.91 last 898.94) total: 64768 /unlink in 84.87 seconds: 763.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 PASS 106 (1368s) Stopping clients: oleg423-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg423-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 oleg423-server: oleg423-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1458 sec ========== 17:20:40 (1755638440) === conf-sanity: start cleanup 17:20:41 (1755638441) === === conf-sanity: finish cleanup 17:20:42 (1755638442) ===