-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 01:21:40 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg207-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg207-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg207-server oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg207-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:22:57 (1761283377) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg207-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32587 331 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3193 (time 1761283425.75 total 10.00 last 319.15) - open/close 5600 (time 1761283435.75 total 20.00 last 240.70) - open/close 7940 (time 1761283445.75 total 30.01 last 233.93) - open/close 10000 (time 1761283455.18 total 39.44 last 218.45) - open/close 12880 (time 1761283465.18 total 49.44 last 287.99) - open/close 15471 (time 1761283475.19 total 59.44 last 259.02) - open/close 17546 (time 1761283485.19 total 69.44 last 207.44) - open/close 19274 (time 1761283495.19 total 79.45 last 172.76) - open/close 20000 (time 1761283499.08 total 83.33 last 186.92) - open/close 22237 (time 1761283509.08 total 93.33 last 223.69) - open/close 24256 (time 1761283519.08 total 103.33 last 201.83) - open/close 26036 (time 1761283529.08 total 113.34 last 177.96) - open/close 28104 (time 1761283539.08 total 123.34 last 206.78) - open/close 30000 (time 1761283548.91 total 133.16 last 192.98) - open/close 32139 (time 1761283558.91 total 143.16 last 213.88) - open/close 35782 (time 1761283568.91 total 153.17 last 364.20) - open/close 39537 (time 1761283578.91 total 163.17 last 375.47) - open/close 43090 (time 1761283588.92 total 173.17 last 355.11) - open/close 46134 (time 1761283598.92 total 183.17 last 304.39) - open/close 49620 (time 1761283608.92 total 193.18 last 348.49) - open/close 51923 (time 1761283618.92 total 203.18 last 230.25) - open/close 54029 (time 1761283628.93 total 213.18 last 210.50) - open/close 56001 (time 1761283638.93 total 223.19 last 197.14) - open/close 58760 (time 1761283648.93 total 233.19 last 275.87) - open/close 60000 (time 1761283653.37 total 237.63 last 279.02) - open/close 63679 (time 1761283663.38 total 247.63 last 367.90) total: 64768 open/close in 250.63 seconds: 258.42 ops/second - /unlink 6815 (time 1761283677.85 total 10.00 last 681.44) - /unlink 10000 (time 1761283682.68 total 14.84 last 658.73) - /unlink 17097 (time 1761283692.68 total 24.84 last 709.61) - /unlink 20000 (time 1761283696.99 total 29.14 last 674.12) - /unlink 27923 (time 1761283706.99 total 39.14 last 792.27) - /unlink 30000 (time 1761283709.60 total 41.75 last 795.66) - /unlink 37738 (time 1761283719.60 total 51.76 last 773.73) - /unlink 40000 (time 1761283722.48 total 54.64 last 785.40) - /unlink 47459 (time 1761283732.48 total 64.64 last 745.82) - /unlink 50000 (time 1761283735.49 total 67.64 last 845.03) - /unlink 58230 (time 1761283745.49 total 77.64 last 823.00) - /unlink 60000 (time 1761283747.92 total 80.07 last 729.79) total: 64768 /unlink in 87.02 seconds: 744.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4764 (time 1761283784.41 total 10.00 last 476.31) - open/close 9506 (time 1761283794.41 total 20.00 last 474.17) - open/close 14085 (time 1761283804.41 total 30.00 last 457.84) - open/close 18778 (time 1761283814.41 total 40.00 last 469.28) - open/close 20000 (time 1761283816.91 total 42.51 last 488.58) - open/close 24210 (time 1761283826.91 total 52.51 last 420.92) - open/close 28560 (time 1761283836.92 total 62.51 last 434.93) - open/close 30000 (time 1761283839.77 total 65.37 last 503.75) - open/close 34683 (time 1761283849.77 total 75.37 last 468.24) - open/close 39239 (time 1761283859.78 total 85.37 last 455.55) - open/close 43980 (time 1761283869.78 total 95.37 last 474.02) - open/close 48803 (time 1761283879.78 total 105.37 last 482.24) - open/close 50000 (time 1761283882.23 total 107.83 last 487.98) - open/close 55042 (time 1761283892.23 total 117.83 last 504.19) - open/close 59734 (time 1761283902.23 total 127.83 last 469.16) total: 64768 open/close in 137.65 seconds: 470.53 ops/second - /unlink 8212 (time 1761283923.40 total 10.00 last 821.17) - /unlink 10000 (time 1761283925.51 total 12.11 last 847.73) - /unlink 17502 (time 1761283935.51 total 22.11 last 750.13) - /unlink 20000 (time 1761283938.82 total 25.42 last 753.94) - /unlink 28008 (time 1761283948.82 total 35.43 last 800.67) - /unlink 30000 (time 1761283951.02 total 37.63 last 904.66) - /unlink 38577 (time 1761283961.02 total 47.63 last 857.69) - /unlink 45951 (time 1761283971.02 total 57.63 last 737.38) - /unlink 50000 (time 1761283976.52 total 63.12 last 737.11) - /unlink 57194 (time 1761283986.52 total 73.12 last 719.35) - /unlink 60000 (time 1761283990.59 total 77.19 last 689.46) total: 64768 /unlink in 83.22 seconds: 778.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4442 (time 1761284026.51 total 10.00 last 444.00) - open/close 8809 (time 1761284036.51 total 20.01 last 436.65) - open/close 10000 (time 1761284039.29 total 22.78 last 429.30) - open/close 14943 (time 1761284049.29 total 32.78 last 494.27) - open/close 20000 (time 1761284058.97 total 42.46 last 522.27) - open/close 25007 (time 1761284068.97 total 52.46 last 500.68) - open/close 29510 (time 1761284078.97 total 62.46 last 450.25) - open/close 34019 (time 1761284088.97 total 72.47 last 450.89) - open/close 38529 (time 1761284098.98 total 82.47 last 450.92) - open/close 40000 (time 1761284102.19 total 85.68 last 458.19) - open/close 44372 (time 1761284112.19 total 95.68 last 437.14) - open/close 48659 (time 1761284122.19 total 105.68 last 428.69) - open/close 50000 (time 1761284125.31 total 108.80 last 429.51) - open/close 54329 (time 1761284135.31 total 118.80 last 432.82) - open/close 58797 (time 1761284145.31 total 128.80 last 446.73) - open/close 60000 (time 1761284148.01 total 131.50 last 445.61) - open/close 64519 (time 1761284158.01 total 141.50 last 451.88) total: 64768 open/close in 142.23 seconds: 455.38 ops/second - /unlink 7100 (time 1761284170.15 total 10.00 last 709.91) - /unlink 10000 (time 1761284173.89 total 13.74 last 775.43) - /unlink 17446 (time 1761284183.89 total 23.74 last 744.59) - /unlink 20000 (time 1761284187.54 total 27.39 last 700.23) - /unlink 26969 (time 1761284197.54 total 37.39 last 696.89) - /unlink 30000 (time 1761284200.88 total 40.73 last 906.71) - /unlink 38157 (time 1761284210.88 total 50.73 last 815.62) - /unlink 40000 (time 1761284213.20 total 53.05 last 796.48) - /unlink 48469 (time 1761284223.21 total 63.06 last 845.52) - /unlink 50000 (time 1761284225.32 total 65.17 last 728.15) - /unlink 58383 (time 1761284235.32 total 75.17 last 838.25) total: 64768 /unlink in 82.17 seconds: 788.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4627 (time 1761284286.16 total 10.00 last 462.67) - open/close 9347 (time 1761284296.16 total 20.00 last 471.92) - open/close 12515 (time 1761284306.17 total 30.01 last 316.58) - open/close 14461 (time 1761284316.17 total 40.01 last 194.53) - open/close 17104 (time 1761284326.17 total 50.02 last 264.21) - open/close 20000 (time 1761284333.89 total 57.74 last 375.03) - open/close 24434 (time 1761284343.90 total 67.74 last 443.27) - open/close 28963 (time 1761284353.90 total 77.74 last 452.82) - open/close 30000 (time 1761284356.21 total 80.05 last 448.95) - open/close 34401 (time 1761284366.21 total 90.05 last 440.07) - open/close 38854 (time 1761284376.21 total 100.06 last 445.22) - open/close 40000 (time 1761284379.16 total 103.01 last 388.42) - open/close 43914 (time 1761284389.16 total 113.01 last 391.37) - open/close 47435 (time 1761284399.16 total 123.01 last 352.07) - open/close 50000 (time 1761284409.01 total 132.85 last 260.55) - open/close 53068 (time 1761284419.01 total 142.85 last 306.72) - open/close 56494 (time 1761284429.01 total 152.86 last 342.53) - open/close 59623 (time 1761284439.02 total 162.86 last 312.79) - open/close 63410 (time 1761284449.02 total 172.86 last 378.68) total: 64768 open/close in 175.82 seconds: 368.37 ops/second - /unlink 7518 (time 1761284463.30 total 10.00 last 751.51) - /unlink 10000 (time 1761284467.10 total 13.80 last 653.21) - /unlink 18231 (time 1761284477.10 total 23.81 last 822.96) - /unlink 20000 (time 1761284479.78 total 26.48 last 661.25) - /unlink 27152 (time 1761284489.78 total 36.48 last 715.15) - /unlink 30000 (time 1761284494.01 total 40.72 last 672.51) - /unlink 37893 (time 1761284504.01 total 50.72 last 789.29) - /unlink 40000 (time 1761284506.47 total 53.17 last 858.79) - /unlink 46103 (time 1761284516.48 total 63.18 last 609.37) - /unlink 47595 (time 1761284526.48 total 73.18 last 149.20) - /unlink 49248 (time 1761284536.48 total 83.19 last 165.28) - /unlink 50000 (time 1761284541.02 total 87.72 last 165.89) - /unlink 51588 (time 1761284551.02 total 97.72 last 158.71) - /unlink 52876 (time 1761284561.02 total 107.73 last 128.79) - /unlink 54006 (time 1761284571.03 total 117.73 last 112.94) - /unlink 55135 (time 1761284581.04 total 127.74 last 112.78) - /unlink 56374 (time 1761284591.04 total 137.74 last 123.88) - /unlink 57975 (time 1761284601.04 total 147.75 last 160.04) - /unlink 59802 (time 1761284611.05 total 157.75 last 182.67) - /unlink 61926 (time 1761284621.05 total 167.75 last 212.34) - /unlink 64025 (time 1761284631.05 total 177.76 last 209.78) total: 64768 /unlink in 182.50 seconds: 354.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 912 (time 1761284663.60 total 10.02 last 91.05) - open/close 1860 (time 1761284673.61 total 20.02 last 94.72) - open/close 2803 (time 1761284683.61 total 30.03 last 94.23) - open/close 3791 (time 1761284693.62 total 40.04 last 98.75) - open/close 4960 (time 1761284703.62 total 50.04 last 116.89) - open/close 5852 (time 1761284713.62 total 60.04 last 89.16) - open/close 6677 (time 1761284723.63 total 70.04 last 82.48) - open/close 7598 (time 1761284733.64 total 80.05 last 92.01) - open/close 8214 (time 1761284743.64 total 90.06 last 61.55) - open/close 8619 (time 1761284753.65 total 100.07 last 40.48) - open/close 8964 (time 1761284763.66 total 110.07 last 34.48) - open/close 9345 (time 1761284773.67 total 120.08 last 38.06) - open/close 9895 (time 1761284783.68 total 130.09 last 54.95) - open/close 10000 (time 1761284785.81 total 132.23 last 49.15) - open/close 10535 (time 1761284795.82 total 142.23 last 53.49) - open/close 11271 (time 1761284805.84 total 152.26 last 73.43) - open/close 12006 (time 1761284815.85 total 162.27 last 73.43) - open/close 12654 (time 1761284825.85 total 172.27 last 64.76) - open/close 13397 (time 1761284835.88 total 182.30 last 74.09) - open/close 14141 (time 1761284845.90 total 192.32 last 74.28) - open/close 14838 (time 1761284855.91 total 202.33 last 69.63) - open/close 15603 (time 1761284865.91 total 212.33 last 76.49) - open/close 16414 (time 1761284875.91 total 222.33 last 81.09) - open/close 17137 (time 1761284885.91 total 232.33 last 72.30) - open/close 17885 (time 1761284895.91 total 242.33 last 74.80) - open/close 18585 (time 1761284905.92 total 252.34 last 69.97) - open/close 19223 (time 1761284915.96 total 262.38 last 63.54) - open/close 19957 (time 1761284925.97 total 272.38 last 73.35) - open/close 20698 (time 1761284935.97 total 282.39 last 74.05) - open/close 21443 (time 1761284945.99 total 292.40 last 74.40) - open/close 22169 (time 1761284955.99 total 302.41 last 72.58) - open/close 22914 (time 1761284966.00 total 312.42 last 74.41) - open/close 23654 (time 1761284976.01 total 322.42 last 73.95) - open/close 24378 (time 1761284986.01 total 332.42 last 72.40) - open/close 25133 (time 1761284996.02 total 342.44 last 75.38) - open/close 25888 (time 1761285006.03 total 352.45 last 75.41) - open/close 26639 (time 1761285016.04 total 362.46 last 75.07) - open/close 27350 (time 1761285026.06 total 372.48 last 70.95) - open/close 28142 (time 1761285036.07 total 382.49 last 79.12) - open/close 28860 (time 1761285046.07 total 392.49 last 71.80) - open/close 29597 (time 1761285056.07 total 402.49 last 73.67) - open/close 30000 (time 1761285061.35 total 407.77 last 76.38) - open/close 30633 (time 1761285071.37 total 417.79 last 63.19) - open/close 31345 (time 1761285081.39 total 427.80 last 71.07) - open/close 32081 (time 1761285091.39 total 437.81 last 73.55) - open/close 32890 (time 1761285101.40 total 447.82 last 80.84) - open/close 33630 (time 1761285111.40 total 457.82 last 73.97) - open/close 34399 (time 1761285121.41 total 467.83 last 76.86) - open/close 35140 (time 1761285131.41 total 477.83 last 74.09) - open/close 35862 (time 1761285141.41 total 487.83 last 72.17) - open/close 36448 (time 1761285151.42 total 497.84 last 58.56) - open/close 37026 (time 1761285161.43 total 507.85 last 57.72) - open/close 37791 (time 1761285171.44 total 517.86 last 76.42) - open/close 38545 (time 1761285181.45 total 527.87 last 75.34) - open/close 39247 (time 1761285191.46 total 537.88 last 70.12) - open/close 39974 (time 1761285201.47 total 547.89 last 72.65) - open/close 40643 (time 1761285211.47 total 557.89 last 66.89) - open/close 41341 (time 1761285221.48 total 567.90 last 69.73) - open/close 42055 (time 1761285231.49 total 577.91 last 71.34) - open/close 42747 (time 1761285241.50 total 587.92 last 69.15) - open/close 43476 (time 1761285251.50 total 597.92 last 72.88) - open/close 44201 (time 1761285261.51 total 607.93 last 72.40) - open/close 44846 (time 1761285271.52 total 617.94 last 64.48) - open/close 45548 (time 1761285281.52 total 627.94 last 70.18) - open/close 46273 (time 1761285291.53 total 637.95 last 72.42) - open/close 46974 (time 1761285301.53 total 647.95 last 70.09) - open/close 47717 (time 1761285311.53 total 657.95 last 74.27) - open/close 48446 (time 1761285321.55 total 667.97 last 72.80) - open/close 49110 (time 1761285331.56 total 677.97 last 66.34) - open/close 49828 (time 1761285341.57 total 687.98 last 71.73) - open/close 50000 (time 1761285344.03 total 690.45 last 69.87) - open/close 50690 (time 1761285354.04 total 700.46 last 68.89) - open/close 51387 (time 1761285364.05 total 710.47 last 69.62) - open/close 52069 (time 1761285374.06 total 720.48 last 68.15) - open/close 52738 (time 1761285384.07 total 730.49 last 66.87) - open/close 53420 (time 1761285394.08 total 740.50 last 68.13) - open/close 54076 (time 1761285404.09 total 750.50 last 65.54) - open/close 54728 (time 1761285414.10 total 760.51 last 65.13) - open/close 55406 (time 1761285424.10 total 770.52 last 67.75) - open/close 56064 (time 1761285434.11 total 780.53 last 65.75) - open/close 56685 (time 1761285444.14 total 790.56 last 61.93) - open/close 57219 (time 1761285454.14 total 800.56 last 53.40) - open/close 57837 (time 1761285464.14 total 810.56 last 61.79) - open/close 58493 (time 1761285474.15 total 820.57 last 65.54) - open/close 59176 (time 1761285484.17 total 830.59 last 68.16) - open/close 59865 (time 1761285494.17 total 840.59 last 68.90) - open/close 60590 (time 1761285504.18 total 850.59 last 72.48) - open/close 61302 (time 1761285514.18 total 860.60 last 71.14) - open/close 62010 (time 1761285524.19 total 870.61 last 70.73) - open/close 62749 (time 1761285534.20 total 880.62 last 73.85) - open/close 63463 (time 1761285544.21 total 890.63 last 71.34) - open/close 64155 (time 1761285554.22 total 900.64 last 69.10) total: 64768 open/close in 909.44 seconds: 71.22 ops/second - /unlink 724 (time 1761285576.58 total 10.01 last 72.34) - /unlink 1420 (time 1761285586.59 total 20.02 last 69.52) - /unlink 2190 (time 1761285596.60 total 30.03 last 76.93) - /unlink 2884 (time 1761285606.61 total 40.04 last 69.33) - /unlink 3635 (time 1761285616.62 total 50.05 last 75.01) - /unlink 4322 (time 1761285626.64 total 60.06 last 68.60) - /unlink 5025 (time 1761285636.64 total 70.07 last 70.29) - /unlink 5744 (time 1761285646.64 total 80.07 last 71.85) - /unlink 6495 (time 1761285656.64 total 90.07 last 75.10) - /unlink 7161 (time 1761285666.65 total 100.08 last 66.59) - /unlink 7889 (time 1761285676.66 total 110.09 last 72.66) - /unlink 8640 (time 1761285686.67 total 120.10 last 75.06) - /unlink 9367 (time 1761285696.68 total 130.11 last 72.63) - /unlink 10000 (time 1761285706.55 total 139.98 last 64.11) - /unlink 10739 (time 1761285716.55 total 149.98 last 73.90) - /unlink 11439 (time 1761285726.56 total 159.99 last 69.95) - /unlink 12165 (time 1761285736.57 total 170.00 last 72.55) - /unlink 12891 (time 1761285746.57 total 180.00 last 72.55) - /unlink 13645 (time 1761285756.58 total 190.01 last 75.36) - /unlink 14274 (time 1761285766.59 total 200.02 last 62.86) - /unlink 14962 (time 1761285776.59 total 210.02 last 68.78) - /unlink 15675 (time 1761285786.60 total 220.03 last 71.19) - /unlink 16372 (time 1761285796.61 total 230.04 last 69.66) - /unlink 17064 (time 1761285806.62 total 240.05 last 69.16) - /unlink 17732 (time 1761285816.63 total 250.06 last 66.72) - /unlink 18444 (time 1761285826.64 total 260.07 last 71.09) - /unlink 19143 (time 1761285836.65 total 270.08 last 69.88) - /unlink 19811 (time 1761285846.65 total 280.08 last 66.78) - /unlink 20000 (time 1761285849.93 total 283.35 last 57.70) - /unlink 20633 (time 1761285859.93 total 293.36 last 63.25) - /unlink 21225 (time 1761285869.93 total 303.36 last 59.19) - /unlink 21911 (time 1761285879.94 total 313.37 last 68.57) - /unlink 22619 (time 1761285889.94 total 323.37 last 70.78) - /unlink 23317 (time 1761285899.95 total 333.38 last 69.75) - /unlink 24050 (time 1761285909.95 total 343.38 last 73.29) - /unlink 24753 (time 1761285919.95 total 353.38 last 70.28) - /unlink 25466 (time 1761285929.96 total 363.39 last 71.23) - /unlink 26192 (time 1761285939.96 total 373.39 last 72.59) - /unlink 26876 (time 1761285949.98 total 383.41 last 68.31) - /unlink 27547 (time 1761285959.98 total 393.41 last 67.09) - /unlink 28306 (time 1761285969.98 total 403.41 last 75.87) - /unlink 28961 (time 1761285979.99 total 413.42 last 65.45) - /unlink 29664 (time 1761285990.00 total 423.43 last 70.20) - /unlink 30000 (time 1761285995.81 total 429.24 last 57.84) - /unlink 30761 (time 1761286005.82 total 439.25 last 76.07) - /unlink 31497 (time 1761286015.82 total 449.25 last 73.57) - /unlink 32222 (time 1761286025.83 total 459.26 last 72.41) - /unlink 33006 (time 1761286035.84 total 469.26 last 78.39) - /unlink 33814 (time 1761286045.84 total 479.27 last 80.77) - /unlink 34595 (time 1761286055.84 total 489.27 last 78.09) - /unlink 35404 (time 1761286065.85 total 499.28 last 80.86) - /unlink 36221 (time 1761286075.85 total 509.28 last 81.67) - /unlink 37042 (time 1761286085.85 total 519.28 last 82.09) - /unlink 37842 (time 1761286095.86 total 529.29 last 79.95) - /unlink 38589 (time 1761286105.88 total 539.31 last 74.55) - /unlink 39341 (time 1761286115.88 total 549.31 last 75.18) - /unlink 40000 (time 1761286123.70 total 557.13 last 84.27) - /unlink 40813 (time 1761286133.70 total 567.13 last 81.29) - /unlink 41602 (time 1761286143.71 total 577.14 last 78.86) - /unlink 42361 (time 1761286153.72 total 587.15 last 75.83) - /unlink 43144 (time 1761286163.72 total 597.15 last 78.24) - /unlink 43949 (time 1761286173.73 total 607.16 last 80.43) - /unlink 44775 (time 1761286183.74 total 617.17 last 82.54) - /unlink 45642 (time 1761286193.75 total 627.18 last 86.63) - /unlink 46516 (time 1761286203.76 total 637.19 last 87.32) - /unlink 47390 (time 1761286213.77 total 647.20 last 87.29) - /unlink 48244 (time 1761286223.77 total 657.20 last 85.37) - /unlink 49083 (time 1761286233.78 total 667.21 last 83.89) - /unlink 49907 (time 1761286243.78 total 677.21 last 82.37) - /unlink 50731 (time 1761286253.79 total 687.22 last 82.34) - /unlink 51581 (time 1761286263.79 total 697.22 last 84.97) - /unlink 52389 (time 1761286273.79 total 707.22 last 80.76) - /unlink 53141 (time 1761286283.80 total 717.23 last 75.19) - /unlink 53958 (time 1761286293.81 total 727.24 last 81.59) - /unlink 54810 (time 1761286303.81 total 737.24 last 85.20) - /unlink 55608 (time 1761286313.81 total 747.24 last 79.78) - /unlink 56431 (time 1761286323.87 total 757.30 last 81.82) - /unlink 57189 (time 1761286333.87 total 767.30 last 75.78) - /unlink 57980 (time 1761286343.88 total 777.31 last 79.05) - /unlink 58809 (time 1761286353.88 total 787.31 last 82.90) - /unlink 59630 (time 1761286363.89 total 797.32 last 82.04) - /unlink 60000 (time 1761286368.67 total 802.10 last 77.32) - /unlink 60809 (time 1761286378.67 total 812.10 last 80.90) - /unlink 61590 (time 1761286388.68 total 822.11 last 78.06) - /unlink 62392 (time 1761286398.68 total 832.11 last 80.19) - /unlink 63215 (time 1761286408.69 total 842.12 last 82.25) - /unlink 64019 (time 1761286418.69 total 852.12 last 80.40) total: 64768 /unlink in 861.60 seconds: 75.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 PASS 106 (3105s) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3238 sec ========== 02:15:39 (1761286539) === conf-sanity: start cleanup 02:15:40 (1761286540) === === conf-sanity: finish cleanup 02:15:42 (1761286542) ===