-----============= acceptance-small: conf-sanity ============----- Thu Jul 31 20:49:23 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg152-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg152-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg152-server oleg152-server: oleg152-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg152-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:50:53 (1754009453) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) oleg152-server: oleg152-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg152-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1023 (time 1754009530.89 total 10.01 last 102.21) - open/close 2016 (time 1754009540.89 total 20.01 last 99.29) - open/close 3292 (time 1754009550.89 total 30.02 last 127.54) - open/close 4477 (time 1754009560.89 total 40.02 last 118.49) - open/close 5731 (time 1754009570.90 total 50.02 last 125.35) - open/close 7030 (time 1754009580.90 total 60.02 last 129.87) - open/close 8701 (time 1754009590.90 total 70.02 last 167.08) - open/close 10000 (time 1754009598.99 total 78.11 last 160.65) - open/close 11910 (time 1754009608.99 total 88.11 last 190.96) - open/close 13752 (time 1754009618.99 total 98.11 last 184.14) - open/close 15500 (time 1754009628.99 total 108.12 last 174.78) - open/close 17797 (time 1754009639.00 total 118.12 last 229.64) - open/close 20000 (time 1754009648.02 total 127.14 last 244.07) - open/close 22608 (time 1754009658.02 total 137.15 last 260.77) - open/close 24830 (time 1754009668.02 total 147.15 last 222.17) - open/close 27464 (time 1754009678.03 total 157.15 last 263.37) - open/close 29889 (time 1754009688.03 total 167.15 last 242.38) - open/close 32226 (time 1754009698.03 total 177.15 last 233.66) - open/close 34573 (time 1754009708.03 total 187.16 last 234.67) - open/close 37792 (time 1754009718.04 total 197.16 last 321.80) - open/close 40000 (time 1754009727.41 total 206.53 last 235.51) - open/close 42177 (time 1754009737.42 total 216.54 last 217.57) - open/close 44083 (time 1754009747.42 total 226.54 last 190.57) - open/close 46281 (time 1754009757.42 total 236.54 last 219.79) - open/close 48728 (time 1754009767.42 total 246.54 last 244.64) - open/close 50000 (time 1754009773.20 total 252.33 last 220.03) - open/close 52615 (time 1754009783.20 total 262.33 last 261.47) - open/close 55455 (time 1754009793.21 total 272.33 last 283.98) - open/close 58979 (time 1754009803.21 total 282.33 last 352.32) - open/close 60000 (time 1754009805.78 total 284.90 last 397.67) - open/close 62950 (time 1754009815.78 total 294.90 last 294.99) total: 64768 open/close in 300.96 seconds: 215.21 ops/second - /unlink 5814 (time 1754009833.24 total 10.00 last 581.34) - /unlink 10000 (time 1754009839.79 total 16.55 last 638.84) - /unlink 16302 (time 1754009849.79 total 26.55 last 630.14) - /unlink 20000 (time 1754009856.21 total 32.97 last 576.43) - /unlink 26027 (time 1754009866.21 total 42.97 last 602.62) - /unlink 30000 (time 1754009872.96 total 49.73 last 588.23) - /unlink 35371 (time 1754009882.97 total 59.73 last 537.04) - /unlink 40000 (time 1754009889.77 total 66.53 last 679.99) - /unlink 46494 (time 1754009899.77 total 76.53 last 649.36) - /unlink 50000 (time 1754009905.11 total 81.87 last 657.50) - /unlink 57656 (time 1754009915.11 total 91.87 last 765.56) - /unlink 60000 (time 1754009918.09 total 94.85 last 784.84) total: 64768 /unlink in 101.38 seconds: 638.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4337 (time 1754009954.86 total 10.00 last 433.63) - open/close 8504 (time 1754009964.86 total 20.00 last 416.67) - open/close 10000 (time 1754009967.98 total 23.12 last 479.70) - open/close 14830 (time 1754009977.98 total 33.12 last 482.98) - open/close 18964 (time 1754009987.98 total 43.12 last 413.30) - open/close 20000 (time 1754009990.43 total 45.58 last 422.31) - open/close 23872 (time 1754010000.43 total 55.58 last 387.18) - open/close 28271 (time 1754010010.43 total 65.58 last 439.87) - open/close 30000 (time 1754010014.45 total 69.59 last 431.03) - open/close 34578 (time 1754010024.45 total 79.59 last 457.79) - open/close 39612 (time 1754010034.45 total 89.59 last 503.36) - open/close 44523 (time 1754010044.45 total 99.59 last 491.01) - open/close 49340 (time 1754010054.45 total 109.59 last 481.68) - open/close 53987 (time 1754010064.45 total 119.59 last 464.62) - open/close 58813 (time 1754010074.45 total 129.59 last 482.60) - open/close 60000 (time 1754010076.95 total 132.10 last 474.14) total: 64768 open/close in 141.92 seconds: 456.37 ops/second - /unlink 7821 (time 1754010098.05 total 10.00 last 782.06) - /unlink 10000 (time 1754010100.42 total 12.38 last 916.32) - /unlink 18168 (time 1754010110.42 total 22.38 last 816.73) - /unlink 20000 (time 1754010112.92 total 24.87 last 734.79) - /unlink 28537 (time 1754010122.92 total 34.87 last 853.63) - /unlink 36864 (time 1754010132.92 total 44.87 last 832.65) - /unlink 40000 (time 1754010137.35 total 49.31 last 707.70) - /unlink 47839 (time 1754010147.35 total 59.31 last 783.85) - /unlink 50000 (time 1754010149.90 total 61.85 last 848.46) - /unlink 57512 (time 1754010159.90 total 71.85 last 751.18) - /unlink 60000 (time 1754010163.19 total 75.15 last 755.25) total: 64768 /unlink in 81.60 seconds: 793.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4168 (time 1754010234.31 total 10.00 last 416.75) - open/close 8490 (time 1754010244.32 total 20.00 last 432.06) - open/close 10000 (time 1754010247.32 total 23.00 last 503.38) - open/close 14856 (time 1754010257.32 total 33.00 last 485.57) - open/close 19937 (time 1754010267.32 total 43.01 last 508.07) - open/close 25092 (time 1754010277.32 total 53.01 last 515.47) - open/close 30000 (time 1754010286.75 total 62.44 last 520.08) - open/close 34504 (time 1754010296.76 total 72.44 last 450.34) - open/close 38923 (time 1754010306.76 total 82.44 last 441.87) - open/close 40000 (time 1754010309.10 total 84.79 last 459.72) - open/close 44457 (time 1754010319.10 total 94.79 last 445.64) - open/close 48695 (time 1754010329.13 total 104.81 last 422.75) - open/close 50000 (time 1754010332.37 total 108.06 last 402.37) - open/close 54160 (time 1754010342.37 total 118.06 last 415.98) - open/close 58158 (time 1754010352.37 total 128.06 last 399.75) - open/close 60000 (time 1754010356.95 total 132.63 last 402.53) - open/close 64164 (time 1754010366.95 total 142.64 last 416.31) total: 64768 open/close in 144.03 seconds: 449.68 ops/second - /unlink 7301 (time 1754010379.76 total 10.00 last 730.03) - /unlink 10000 (time 1754010383.60 total 13.84 last 703.61) - /unlink 16866 (time 1754010393.60 total 23.84 last 686.56) - /unlink 20000 (time 1754010398.01 total 28.25 last 709.83) - /unlink 27105 (time 1754010408.01 total 38.25 last 710.46) - /unlink 30000 (time 1754010411.69 total 41.93 last 787.33) - /unlink 37456 (time 1754010421.69 total 51.93 last 745.55) - /unlink 40000 (time 1754010425.19 total 55.43 last 727.85) - /unlink 47355 (time 1754010435.19 total 65.43 last 735.46) - /unlink 50000 (time 1754010439.05 total 69.29 last 685.37) - /unlink 57203 (time 1754010449.05 total 79.29 last 720.22) - /unlink 60000 (time 1754010452.82 total 83.06 last 741.75) total: 64768 /unlink in 89.59 seconds: 722.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4799 (time 1754010489.16 total 10.00 last 479.86) - open/close 9459 (time 1754010499.16 total 20.00 last 465.95) - open/close 14083 (time 1754010509.20 total 30.04 last 460.61) - open/close 18738 (time 1754010519.20 total 40.04 last 465.43) - open/close 20000 (time 1754010521.94 total 42.78 last 460.48) - open/close 24621 (time 1754010531.94 total 52.78 last 462.03) - open/close 29194 (time 1754010541.94 total 62.78 last 457.28) - open/close 33722 (time 1754010551.95 total 72.78 last 452.79) - open/close 38352 (time 1754010561.95 total 82.79 last 462.89) - open/close 40000 (time 1754010565.45 total 86.29 last 470.77) - open/close 44448 (time 1754010575.45 total 96.29 last 444.78) - open/close 48924 (time 1754010585.45 total 106.29 last 447.57) - open/close 50000 (time 1754010587.99 total 108.83 last 424.16) - open/close 54777 (time 1754010597.99 total 118.83 last 477.63) - open/close 59503 (time 1754010607.99 total 128.83 last 472.49) - open/close 63836 (time 1754010617.99 total 138.83 last 433.24) total: 64768 open/close in 140.97 seconds: 459.46 ops/second - /unlink 7610 (time 1754010631.49 total 10.00 last 760.95) - /unlink 10000 (time 1754010634.95 total 13.46 last 690.33) - /unlink 17293 (time 1754010644.95 total 23.46 last 729.18) - /unlink 20000 (time 1754010648.27 total 26.78 last 815.32) - /unlink 27292 (time 1754010658.27 total 36.79 last 729.16) - /unlink 30000 (time 1754010662.02 total 40.53 last 723.27) - /unlink 37281 (time 1754010672.02 total 50.53 last 728.04) - /unlink 40000 (time 1754010675.97 total 54.48 last 689.16) - /unlink 47498 (time 1754010685.97 total 64.48 last 749.79) - /unlink 50000 (time 1754010689.21 total 67.72 last 771.08) - /unlink 57382 (time 1754010699.21 total 77.72 last 738.05) - /unlink 60000 (time 1754010702.86 total 81.37 last 718.22) total: 64768 /unlink in 87.97 seconds: 736.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4624 (time 1754010738.63 total 10.00 last 462.38) - open/close 9335 (time 1754010748.64 total 20.00 last 471.02) - open/close 13965 (time 1754010758.64 total 30.00 last 462.91) - open/close 18618 (time 1754010768.64 total 40.01 last 465.26) - open/close 20000 (time 1754010771.68 total 43.05 last 454.31) - open/close 24624 (time 1754010781.68 total 53.05 last 462.30) - open/close 29231 (time 1754010791.68 total 63.05 last 460.69) - open/close 33864 (time 1754010801.68 total 73.05 last 463.27) - open/close 38456 (time 1754010811.69 total 83.05 last 459.11) - open/close 40000 (time 1754010814.95 total 86.32 last 472.72) - open/close 44674 (time 1754010824.95 total 96.32 last 467.28) - open/close 49045 (time 1754010834.96 total 106.32 last 437.02) - open/close 50000 (time 1754010837.18 total 108.54 last 429.90) - open/close 54394 (time 1754010847.18 total 118.54 last 439.37) - open/close 58939 (time 1754010857.18 total 128.54 last 454.50) - open/close 60000 (time 1754010859.63 total 130.99 last 433.05) - open/close 64522 (time 1754010869.63 total 141.00 last 452.14) total: 64768 open/close in 141.62 seconds: 457.34 ops/second - /unlink 7838 (time 1754010881.56 total 10.00 last 783.79) - /unlink 10000 (time 1754010884.26 total 12.70 last 801.29) - /unlink 17323 (time 1754010894.26 total 22.70 last 732.24) - /unlink 20000 (time 1754010897.52 total 25.96 last 821.38) - /unlink 26446 (time 1754010907.52 total 35.96 last 644.59) - /unlink 30000 (time 1754010912.27 total 40.71 last 748.19) - /unlink 37389 (time 1754010922.27 total 50.71 last 738.78) - /unlink 40000 (time 1754010925.50 total 53.94 last 807.80) - /unlink 47548 (time 1754010935.50 total 63.94 last 754.76) - /unlink 50000 (time 1754010938.81 total 67.26 last 740.07) - /unlink 58268 (time 1754010948.81 total 77.26 last 826.70) - /unlink 60000 (time 1754010951.15 total 79.60 last 740.21) total: 64768 /unlink in 85.58 seconds: 756.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 PASS 106 (1546s) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 oleg152-server: oleg152-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1659 sec ========== 21:17:03 (1754011023) === conf-sanity: start cleanup 21:17:04 (1754011024) === === conf-sanity: finish cleanup 21:17:05 (1754011025) ===