-----============= acceptance-small: conf-sanity ============----- Fri Dec 5 07:20:06 EST 2025 oleg238-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg238-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg238-server oleg238-server: oleg238-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg238-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg238-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:22:26 (1764937346) Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f) oleg238-server: oleg238-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg238-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg238-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 738 (time 1764937456.16 total 10.01 last 73.73) - open/close 1539 (time 1764937466.17 total 20.02 last 80.02) - open/close 2242 (time 1764937476.18 total 30.03 last 70.25) - open/close 2971 (time 1764937486.19 total 40.03 last 72.83) - open/close 3700 (time 1764937496.19 total 50.03 last 72.90) - open/close 4416 (time 1764937506.19 total 60.04 last 71.59) - open/close 5159 (time 1764937516.20 total 70.05 last 74.22) - open/close 5895 (time 1764937526.20 total 80.05 last 73.57) - open/close 6592 (time 1764937536.21 total 90.06 last 69.65) - open/close 7299 (time 1764937546.22 total 100.06 last 70.67) - open/close 8041 (time 1764937556.22 total 110.07 last 74.16) - open/close 8731 (time 1764937566.23 total 120.08 last 68.95) - open/close 9442 (time 1764937576.25 total 130.10 last 70.93) - open/close 10000 (time 1764937583.14 total 136.99 last 81.02) - open/close 10750 (time 1764937593.15 total 146.99 last 74.95) - open/close 11436 (time 1764937603.16 total 157.00 last 68.53) - open/close 12133 (time 1764937613.16 total 167.01 last 69.68) - open/close 12818 (time 1764937623.17 total 177.02 last 68.42) - open/close 13521 (time 1764937633.18 total 187.03 last 70.25) - open/close 14306 (time 1764937643.18 total 197.03 last 78.49) - open/close 15061 (time 1764937653.19 total 207.04 last 75.42) - open/close 15752 (time 1764937663.20 total 217.05 last 69.02) - open/close 16436 (time 1764937673.20 total 227.05 last 68.39) - open/close 17152 (time 1764937683.21 total 237.06 last 71.53) - open/close 17880 (time 1764937693.22 total 247.07 last 72.76) - open/close 18677 (time 1764937703.22 total 257.07 last 79.69) - open/close 19559 (time 1764937713.22 total 267.07 last 88.20) - open/close 20000 (time 1764937718.17 total 272.02 last 89.07) - open/close 20778 (time 1764937728.17 total 282.02 last 77.80) - open/close 21604 (time 1764937738.18 total 292.02 last 82.56) - open/close 22423 (time 1764937748.18 total 302.03 last 81.88) - open/close 23215 (time 1764937758.18 total 312.03 last 79.19) - open/close 23985 (time 1764937768.19 total 322.04 last 76.89) - open/close 24817 (time 1764937778.20 total 332.05 last 83.13) - open/close 25609 (time 1764937788.21 total 342.06 last 79.13) - open/close 26425 (time 1764937798.22 total 352.07 last 81.51) - open/close 27316 (time 1764937808.23 total 362.08 last 89.02) - open/close 28137 (time 1764937818.23 total 372.08 last 82.09) - open/close 28946 (time 1764937828.24 total 382.09 last 80.84) - open/close 29675 (time 1764937838.24 total 392.09 last 72.87) - open/close 30000 (time 1764937843.08 total 396.93 last 67.20) - open/close 30807 (time 1764937853.08 total 406.93 last 80.69) - open/close 31649 (time 1764937863.09 total 416.94 last 84.10) - open/close 32475 (time 1764937873.10 total 426.95 last 82.54) - open/close 33300 (time 1764937883.11 total 436.96 last 82.45) - open/close 34031 (time 1764937893.11 total 446.96 last 73.09) - open/close 34722 (time 1764937903.11 total 456.96 last 69.07) - open/close 35390 (time 1764937913.11 total 466.96 last 66.80) - open/close 36158 (time 1764937923.12 total 476.97 last 76.71) - open/close 36942 (time 1764937933.13 total 486.98 last 78.33) - open/close 37760 (time 1764937943.14 total 496.99 last 81.75) - open/close 38611 (time 1764937953.14 total 506.99 last 85.07) - open/close 39452 (time 1764937963.16 total 517.01 last 83.94) - open/close 40000 (time 1764937969.79 total 523.64 last 82.68) - open/close 40834 (time 1764937979.79 total 533.64 last 83.38) - open/close 41677 (time 1764937989.80 total 543.64 last 84.27) - open/close 42553 (time 1764937999.81 total 553.66 last 87.47) - open/close 43357 (time 1764938009.82 total 563.66 last 80.36) - open/close 44134 (time 1764938019.82 total 573.67 last 77.68) - open/close 44985 (time 1764938029.82 total 583.67 last 85.08) - open/close 45731 (time 1764938039.82 total 593.67 last 74.58) - open/close 46560 (time 1764938049.83 total 603.68 last 82.85) - open/close 47363 (time 1764938059.83 total 613.68 last 80.27) - open/close 48161 (time 1764938069.84 total 623.68 last 79.78) - open/close 48986 (time 1764938079.84 total 633.68 last 82.50) - open/close 49822 (time 1764938089.84 total 643.69 last 83.59) - open/close 50000 (time 1764938091.90 total 645.75 last 86.41) - open/close 50798 (time 1764938101.91 total 655.76 last 79.68) - open/close 51612 (time 1764938111.92 total 665.76 last 81.38) - open/close 52477 (time 1764938121.92 total 675.77 last 86.46) - open/close 53306 (time 1764938131.92 total 685.77 last 82.86) - open/close 54127 (time 1764938141.93 total 695.78 last 82.06) - open/close 54896 (time 1764938151.95 total 705.80 last 76.73) - open/close 55719 (time 1764938161.95 total 715.80 last 82.29) - open/close 56520 (time 1764938171.96 total 725.81 last 80.03) - open/close 57319 (time 1764938181.97 total 735.81 last 79.86) - open/close 58147 (time 1764938191.97 total 745.82 last 82.79) - open/close 59009 (time 1764938201.98 total 755.83 last 86.11) - open/close 59880 (time 1764938211.99 total 765.84 last 86.99) - open/close 60930 (time 1764938222.00 total 775.85 last 104.90) - open/close 61940 (time 1764938232.01 total 785.86 last 100.87) - open/close 62716 (time 1764938242.02 total 795.87 last 77.56) - open/close 63527 (time 1764938252.07 total 805.91 last 80.70) - open/close 64253 (time 1764938262.07 total 815.92 last 72.56) total: 64768 open/close in 822.46 seconds: 78.75 ops/second - /unlink 1215 (time 1764938281.98 total 10.00 last 121.46) - /unlink 2498 (time 1764938291.98 total 20.01 last 128.26) - /unlink 3803 (time 1764938301.99 total 30.02 last 130.36) - /unlink 5132 (time 1764938312.00 total 40.02 last 132.80) - /unlink 6509 (time 1764938322.00 total 50.02 last 137.69) - /unlink 8006 (time 1764938332.00 total 60.03 last 149.67) - /unlink 9452 (time 1764938342.00 total 70.03 last 144.56) - /unlink 10000 (time 1764938346.21 total 74.23 last 130.36) - /unlink 11471 (time 1764938356.22 total 84.24 last 146.97) - /unlink 12966 (time 1764938366.22 total 94.24 last 149.47) - /unlink 14288 (time 1764938376.22 total 104.25 last 132.18) - /unlink 15581 (time 1764938386.23 total 114.25 last 129.24) - /unlink 16836 (time 1764938396.23 total 124.26 last 125.39) - /unlink 18094 (time 1764938406.24 total 134.26 last 125.77) - /unlink 19426 (time 1764938416.24 total 144.26 last 133.16) - /unlink 20000 (time 1764938420.58 total 148.60 last 132.33) - /unlink 21288 (time 1764938430.58 total 158.61 last 128.76) - /unlink 22605 (time 1764938440.58 total 168.61 last 131.69) - /unlink 24114 (time 1764938450.59 total 178.62 last 150.76) - /unlink 25450 (time 1764938460.59 total 188.62 last 133.57) - /unlink 26652 (time 1764938470.60 total 198.62 last 120.16) - /unlink 27910 (time 1764938480.60 total 208.62 last 125.79) - /unlink 29233 (time 1764938490.60 total 218.63 last 132.25) - /unlink 30000 (time 1764938496.98 total 225.01 last 120.16) - /unlink 31228 (time 1764938506.99 total 235.01 last 122.73) - /unlink 32507 (time 1764938516.99 total 245.02 last 127.86) - /unlink 33717 (time 1764938527.00 total 255.02 last 120.95) - /unlink 34922 (time 1764938537.00 total 265.03 last 120.44) - /unlink 36138 (time 1764938547.01 total 275.03 last 121.53) - /unlink 37447 (time 1764938557.01 total 285.03 last 130.89) - /unlink 38761 (time 1764938567.01 total 295.04 last 131.34) - /unlink 40000 (time 1764938575.64 total 303.67 last 143.59) - /unlink 41309 (time 1764938585.65 total 313.67 last 130.82) - /unlink 42675 (time 1764938595.66 total 323.68 last 136.48) - /unlink 43916 (time 1764938605.68 total 333.70 last 123.82) - /unlink 45182 (time 1764938615.68 total 343.70 last 126.60) - /unlink 46489 (time 1764938625.68 total 353.71 last 130.63) - /unlink 47790 (time 1764938635.69 total 363.71 last 130.07) - /unlink 49108 (time 1764938645.69 total 373.72 last 131.74) - /unlink 50000 (time 1764938652.59 total 380.62 last 129.23) - /unlink 51284 (time 1764938662.60 total 390.63 last 128.30) - /unlink 52704 (time 1764938672.61 total 400.64 last 141.87) - /unlink 54017 (time 1764938682.61 total 410.64 last 131.26) - /unlink 55382 (time 1764938692.64 total 420.67 last 136.11) - /unlink 56694 (time 1764938702.65 total 430.68 last 131.08) - /unlink 58010 (time 1764938712.66 total 440.68 last 131.55) - /unlink 59267 (time 1764938722.67 total 450.69 last 125.53) - /unlink 60000 (time 1764938728.75 total 456.78 last 120.51) - /unlink 61319 (time 1764938738.77 total 466.79 last 131.66) - /unlink 62572 (time 1764938748.78 total 476.81 last 125.11) - /unlink 63893 (time 1764938758.79 total 486.81 last 132.06) total: 64768 /unlink in 494.16 seconds: 131.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 883 (time 1764938800.65 total 10.01 last 88.19) - open/close 1764 (time 1764938810.65 total 20.02 last 88.07) - open/close 2659 (time 1764938820.66 total 30.03 last 89.39) - open/close 3530 (time 1764938830.67 total 40.04 last 87.02) - open/close 4338 (time 1764938840.68 total 50.05 last 80.74) - open/close 5171 (time 1764938850.69 total 60.05 last 83.23) - open/close 6046 (time 1764938860.69 total 70.06 last 87.47) - open/close 6836 (time 1764938870.72 total 80.08 last 78.80) - open/close 7551 (time 1764938880.72 total 90.08 last 71.50) - open/close 8325 (time 1764938890.73 total 100.09 last 77.36) - open/close 9184 (time 1764938900.73 total 110.09 last 85.87) - open/close 10000 (time 1764938910.10 total 119.46 last 87.10) - open/close 10906 (time 1764938920.11 total 129.47 last 90.52) - open/close 11810 (time 1764938930.12 total 139.49 last 90.24) - open/close 12743 (time 1764938940.14 total 149.51 last 93.13) - open/close 13705 (time 1764938950.15 total 159.51 last 96.13) - open/close 14530 (time 1764938960.15 total 169.52 last 82.48) - open/close 15342 (time 1764938970.15 total 179.52 last 81.18) - open/close 16125 (time 1764938980.17 total 189.53 last 78.21) - open/close 16912 (time 1764938990.17 total 199.53 last 78.69) - open/close 17776 (time 1764939000.17 total 209.54 last 86.35) - open/close 18561 (time 1764939010.18 total 219.54 last 78.47) - open/close 19439 (time 1764939020.19 total 229.55 last 87.71) - open/close 20000 (time 1764939026.89 total 236.25 last 83.72) - open/close 20847 (time 1764939036.89 total 246.26 last 84.65) - open/close 21667 (time 1764939046.90 total 256.26 last 81.96) - open/close 22459 (time 1764939056.93 total 266.29 last 78.98) - open/close 23234 (time 1764939066.94 total 276.30 last 77.40) - open/close 24034 (time 1764939076.94 total 286.31 last 79.99) - open/close 24837 (time 1764939086.96 total 296.32 last 80.19) - open/close 25661 (time 1764939096.97 total 306.33 last 82.32) - open/close 26454 (time 1764939106.97 total 316.33 last 79.28) - open/close 27336 (time 1764939116.98 total 326.34 last 88.11) - open/close 28157 (time 1764939127.00 total 336.36 last 81.95) - open/close 28974 (time 1764939137.00 total 346.36 last 81.67) - open/close 29808 (time 1764939147.01 total 356.37 last 83.32) - open/close 30000 (time 1764939149.46 total 358.83 last 78.25) - open/close 30823 (time 1764939159.47 total 368.83 last 82.24) - open/close 31654 (time 1764939169.48 total 378.84 last 83.05) - open/close 32424 (time 1764939179.48 total 388.84 last 76.99) - open/close 33234 (time 1764939189.49 total 398.85 last 80.91) - open/close 34021 (time 1764939199.50 total 408.86 last 78.64) - open/close 34852 (time 1764939209.50 total 418.86 last 83.09) - open/close 35729 (time 1764939219.50 total 428.87 last 87.65) - open/close 36588 (time 1764939229.51 total 438.87 last 85.85) - open/close 37375 (time 1764939239.52 total 448.88 last 78.65) - open/close 38222 (time 1764939249.52 total 458.88 last 84.67) - open/close 39089 (time 1764939259.53 total 468.89 last 86.61) - open/close 39951 (time 1764939269.53 total 478.90 last 86.17) - open/close 40854 (time 1764939279.54 total 488.91 last 90.22) - open/close 41683 (time 1764939289.54 total 498.91 last 82.88) - open/close 42608 (time 1764939299.55 total 508.91 last 92.46) - open/close 43434 (time 1764939309.55 total 518.91 last 82.59) - open/close 44232 (time 1764939319.55 total 528.92 last 79.76) - open/close 45077 (time 1764939329.56 total 538.92 last 84.47) - open/close 45933 (time 1764939339.57 total 548.93 last 85.51) - open/close 46877 (time 1764939349.57 total 558.94 last 94.36) - open/close 47752 (time 1764939359.58 total 568.94 last 87.49) - open/close 48623 (time 1764939369.58 total 578.94 last 87.08) - open/close 49429 (time 1764939379.59 total 588.95 last 80.49) - open/close 50000 (time 1764939386.27 total 595.63 last 85.48) - open/close 50880 (time 1764939396.28 total 605.64 last 87.92) - open/close 51762 (time 1764939406.29 total 615.65 last 88.12) - open/close 52615 (time 1764939416.29 total 625.66 last 85.26) - open/close 53523 (time 1764939426.30 total 635.67 last 90.73) - open/close 54394 (time 1764939436.31 total 645.68 last 87.00) - open/close 55260 (time 1764939446.32 total 655.69 last 86.51) - open/close 56230 (time 1764939456.33 total 665.69 last 96.96) - open/close 57111 (time 1764939466.33 total 675.69 last 88.08) - open/close 57867 (time 1764939476.33 total 685.70 last 75.58) - open/close 58685 (time 1764939486.33 total 695.70 last 81.79) - open/close 59519 (time 1764939496.34 total 705.70 last 83.35) - open/close 60000 (time 1764939502.27 total 711.63 last 81.13) - open/close 60734 (time 1764939512.27 total 721.64 last 73.37) - open/close 61502 (time 1764939522.28 total 731.64 last 76.73) - open/close 62320 (time 1764939532.29 total 741.65 last 81.75) - open/close 63191 (time 1764939542.34 total 751.70 last 86.64) - open/close 63956 (time 1764939552.35 total 761.71 last 76.44) - open/close 64761 (time 1764939562.36 total 771.72 last 80.44) total: 64768 open/close in 771.79 seconds: 83.92 ops/second - /unlink 1142 (time 1764939575.64 total 10.00 last 114.16) - /unlink 2308 (time 1764939585.64 total 20.01 last 116.56) - /unlink 3494 (time 1764939595.65 total 30.01 last 118.53) - /unlink 4786 (time 1764939605.65 total 40.02 last 129.17) - /unlink 6204 (time 1764939615.65 total 50.02 last 141.75) - /unlink 7727 (time 1764939625.66 total 60.03 last 152.17) - /unlink 9305 (time 1764939635.66 total 70.03 last 157.71) - /unlink 10000 (time 1764939640.23 total 74.60 last 152.13) - /unlink 11423 (time 1764939650.24 total 84.61 last 142.24) - /unlink 12912 (time 1764939660.24 total 94.61 last 148.84) - /unlink 14213 (time 1764939670.24 total 104.61 last 130.07) - /unlink 15768 (time 1764939680.25 total 114.62 last 155.41) - /unlink 17412 (time 1764939690.26 total 124.63 last 164.28) - /unlink 18732 (time 1764939700.26 total 134.63 last 131.95) - /unlink 20000 (time 1764939708.56 total 142.93 last 152.75) - /unlink 21418 (time 1764939718.57 total 152.94 last 141.61) - /unlink 22917 (time 1764939728.58 total 162.94 last 149.88) - /unlink 24387 (time 1764939738.58 total 172.95 last 146.98) - /unlink 25903 (time 1764939748.58 total 182.95 last 151.56) - /unlink 27435 (time 1764939758.59 total 192.96 last 153.00) - /unlink 29185 (time 1764939768.60 total 202.97 last 174.87) - /unlink 30000 (time 1764939773.58 total 207.95 last 163.68) - /unlink 31565 (time 1764939783.59 total 217.96 last 156.34) - /unlink 32940 (time 1764939793.59 total 227.96 last 137.47) - /unlink 34250 (time 1764939803.60 total 237.97 last 130.92) - /unlink 35537 (time 1764939813.61 total 247.98 last 128.57) - /unlink 37213 (time 1764939823.61 total 257.98 last 167.54) - /unlink 38923 (time 1764939833.61 total 267.98 last 170.95) - /unlink 40000 (time 1764939840.14 total 274.51 last 164.99) - /unlink 41562 (time 1764939850.14 total 284.51 last 156.17) - /unlink 43265 (time 1764939860.15 total 294.52 last 170.21) - /unlink 45272 (time 1764939870.15 total 304.52 last 200.70) - /unlink 47147 (time 1764939880.15 total 314.52 last 187.47) - /unlink 48714 (time 1764939890.15 total 324.52 last 156.67) - /unlink 50000 (time 1764939897.65 total 332.02 last 171.51) - /unlink 51817 (time 1764939907.65 total 342.02 last 181.64) - /unlink 53845 (time 1764939917.66 total 352.02 last 202.78) - /unlink 55568 (time 1764939927.66 total 362.03 last 172.18) - /unlink 57046 (time 1764939937.67 total 372.04 last 147.73) - /unlink 58621 (time 1764939947.67 total 382.04 last 157.48) - /unlink 60000 (time 1764939955.77 total 390.14 last 170.16) - /unlink 61629 (time 1764939965.78 total 400.14 last 162.84) - /unlink 63106 (time 1764939975.79 total 410.16 last 147.51) - /unlink 64666 (time 1764939985.79 total 420.16 last 155.95) total: 64768 /unlink in 420.65 seconds: 153.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1065 (time 1764940017.76 total 10.00 last 106.47) - open/close 2058 (time 1764940027.76 total 20.01 last 99.26) - open/close 3087 (time 1764940037.78 total 30.02 last 102.76) - open/close 4040 (time 1764940047.78 total 40.02 last 95.26) - open/close 4983 (time 1764940057.78 total 50.03 last 94.27) - open/close 5871 (time 1764940067.79 total 60.04 last 88.72) - open/close 6855 (time 1764940077.79 total 70.04 last 98.38) - open/close 7652 (time 1764940087.80 total 80.04 last 79.66) - open/close 8547 (time 1764940097.82 total 90.06 last 89.33) - open/close 9448 (time 1764940107.82 total 100.06 last 90.10) - open/close 10000 (time 1764940113.30 total 105.55 last 100.65) - open/close 11080 (time 1764940123.31 total 115.56 last 107.89) - open/close 12262 (time 1764940133.31 total 125.56 last 118.19) - open/close 13508 (time 1764940143.32 total 135.56 last 124.51) - open/close 14668 (time 1764940153.32 total 145.57 last 115.98) - open/close 15896 (time 1764940163.35 total 155.59 last 122.47) - open/close 17279 (time 1764940173.35 total 165.59 last 138.28) - open/close 18700 (time 1764940183.35 total 175.60 last 142.06) - open/close 19981 (time 1764940193.36 total 185.60 last 128.04) - open/close 21507 (time 1764940203.36 total 195.60 last 152.58) - open/close 23197 (time 1764940213.36 total 205.60 last 168.97) - open/close 24392 (time 1764940223.37 total 215.62 last 119.36) - open/close 25607 (time 1764940233.37 total 225.62 last 121.48) - open/close 26685 (time 1764940243.37 total 235.62 last 107.80) - open/close 27822 (time 1764940253.39 total 245.63 last 113.57) - open/close 28880 (time 1764940263.39 total 255.63 last 105.73) - open/close 30000 (time 1764940272.28 total 264.52 last 126.04) - open/close 31251 (time 1764940282.29 total 274.53 last 124.98) - open/close 32405 (time 1764940292.29 total 284.53 last 115.36) - open/close 33397 (time 1764940302.30 total 294.54 last 99.09) - open/close 34603 (time 1764940312.31 total 304.55 last 120.52) - open/close 35874 (time 1764940322.32 total 314.56 last 127.01) - open/close 37027 (time 1764940332.32 total 324.56 last 115.23) - open/close 38118 (time 1764940342.32 total 334.57 last 109.09) - open/close 39152 (time 1764940352.32 total 344.57 last 103.40) - open/close 40000 (time 1764940359.64 total 351.88 last 115.93) - open/close 41216 (time 1764940369.64 total 361.88 last 121.55) - open/close 42254 (time 1764940379.64 total 371.89 last 103.79) - open/close 43265 (time 1764940389.65 total 381.89 last 101.02) - open/close 44537 (time 1764940399.65 total 391.90 last 127.14) - open/close 45740 (time 1764940409.66 total 401.90 last 120.24) - open/close 46988 (time 1764940419.66 total 411.91 last 124.75) - open/close 48460 (time 1764940429.67 total 421.91 last 147.18) - open/close 49662 (time 1764940439.67 total 431.91 last 120.16) - open/close 50000 (time 1764940442.17 total 434.42 last 134.90) - open/close 51141 (time 1764940452.17 total 444.42 last 114.09) - open/close 52206 (time 1764940462.18 total 454.43 last 106.40) - open/close 53404 (time 1764940472.19 total 464.43 last 119.74) - open/close 54730 (time 1764940482.20 total 474.44 last 132.46) - open/close 56063 (time 1764940492.20 total 484.45 last 133.26) - open/close 57381 (time 1764940502.21 total 494.45 last 131.76) - open/close 58859 (time 1764940512.21 total 504.46 last 147.68) - open/close 60000 (time 1764940521.77 total 514.02 last 119.35) - open/close 61288 (time 1764940531.78 total 524.02 last 128.77) - open/close 62645 (time 1764940541.78 total 534.02 last 135.68) - open/close 63735 (time 1764940551.78 total 544.02 last 108.98) total: 64768 open/close in 552.79 seconds: 117.17 ops/second - /unlink 2443 (time 1764940572.50 total 10.00 last 244.27) - /unlink 4963 (time 1764940582.50 total 20.00 last 251.92) - /unlink 7277 (time 1764940592.50 total 30.01 last 231.36) - /unlink 10000 (time 1764940602.39 total 39.89 last 275.40) - /unlink 12798 (time 1764940612.39 total 49.90 last 279.67) - /unlink 15829 (time 1764940622.39 total 59.90 last 303.07) - /unlink 18859 (time 1764940632.40 total 69.90 last 302.95) - /unlink 20000 (time 1764940635.54 total 73.05 last 362.71) - /unlink 23849 (time 1764940645.54 total 83.05 last 384.82) - /unlink 27048 (time 1764940655.55 total 93.05 last 319.84) - /unlink 30000 (time 1764940664.69 total 102.20 last 322.75) - /unlink 33189 (time 1764940674.69 total 112.20 last 318.87) - /unlink 36513 (time 1764940684.69 total 122.20 last 332.37) - /unlink 39873 (time 1764940694.69 total 132.20 last 335.98) - /unlink 44061 (time 1764940704.69 total 142.20 last 418.79) - /unlink 47358 (time 1764940714.70 total 152.20 last 329.59) - /unlink 50000 (time 1764940723.28 total 160.79 last 307.68) - /unlink 53753 (time 1764940733.29 total 170.79 last 375.22) - /unlink 57678 (time 1764940743.29 total 180.79 last 392.40) - /unlink 60000 (time 1764940750.74 total 188.24 last 311.65) - /unlink 62780 (time 1764940760.74 total 198.25 last 277.97) total: 64768 /unlink in 205.50 seconds: 315.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2767 (time 1764940794.72 total 10.00 last 276.66) - open/close 5127 (time 1764940804.72 total 20.00 last 235.96) - open/close 7608 (time 1764940814.72 total 30.00 last 248.09) - open/close 9829 (time 1764940824.72 total 40.01 last 221.95) - open/close 11844 (time 1764940834.73 total 50.02 last 201.40) - open/close 13814 (time 1764940844.73 total 60.02 last 196.92) - open/close 15960 (time 1764940854.74 total 70.02 last 214.56) - open/close 17608 (time 1764940864.74 total 80.02 last 164.74) - open/close 20000 (time 1764940874.49 total 89.77 last 245.41) - open/close 22431 (time 1764940884.49 total 99.78 last 243.01) - open/close 24281 (time 1764940894.49 total 109.78 last 184.98) - open/close 26696 (time 1764940904.49 total 119.78 last 241.48) - open/close 29601 (time 1764940914.49 total 129.78 last 290.48) - open/close 32546 (time 1764940924.49 total 139.78 last 294.47) - open/close 35078 (time 1764940934.49 total 149.78 last 253.16) - open/close 37748 (time 1764940944.50 total 159.78 last 266.98) - open/close 39868 (time 1764940954.50 total 169.78 last 211.96) - open/close 42114 (time 1764940964.50 total 179.79 last 224.51) - open/close 43991 (time 1764940974.51 total 189.79 last 187.60) - open/close 45636 (time 1764940984.51 total 199.79 last 164.47) - open/close 47667 (time 1764940994.51 total 209.80 last 203.05) - open/close 49193 (time 1764941004.51 total 219.80 last 152.59) - open/close 50000 (time 1764941010.25 total 225.54 last 140.59) - open/close 51691 (time 1764941020.25 total 235.54 last 169.10) - open/close 53681 (time 1764941030.26 total 245.54 last 198.93) - open/close 55580 (time 1764941040.26 total 255.54 last 189.85) - open/close 57097 (time 1764941050.26 total 265.54 last 151.69) - open/close 58782 (time 1764941060.26 total 275.55 last 168.42) - open/close 60000 (time 1764941066.40 total 281.68 last 198.62) - open/close 62037 (time 1764941076.40 total 291.69 last 203.60) - open/close 63742 (time 1764941086.41 total 301.69 last 170.43) total: 64768 open/close in 306.80 seconds: 211.11 ops/second - /unlink 3214 (time 1764941103.31 total 10.00 last 321.25) - /unlink 6814 (time 1764941113.32 total 20.01 last 359.68) - /unlink 10000 (time 1764941119.41 total 26.11 last 522.96) - /unlink 16152 (time 1764941129.41 total 36.11 last 615.16) - /unlink 20000 (time 1764941139.20 total 45.90 last 393.10) - /unlink 23893 (time 1764941149.20 total 55.90 last 389.24) - /unlink 28402 (time 1764941159.22 total 65.91 last 450.33) - /unlink 30000 (time 1764941162.66 total 69.36 last 463.55) - /unlink 34516 (time 1764941172.66 total 79.36 last 451.57) - /unlink 39357 (time 1764941182.66 total 89.36 last 484.08) - /unlink 43872 (time 1764941192.66 total 99.36 last 451.47) - /unlink 48854 (time 1764941202.66 total 109.36 last 498.19) - /unlink 54512 (time 1764941212.67 total 119.36 last 565.76) - /unlink 59250 (time 1764941222.67 total 129.36 last 473.73) - /unlink 64183 (time 1764941232.67 total 139.36 last 493.27) total: 64768 /unlink in 140.81 seconds: 459.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2285 (time 1764941261.05 total 10.00 last 228.46) - open/close 4501 (time 1764941271.05 total 20.01 last 221.52) - open/close 6930 (time 1764941281.05 total 30.01 last 242.83) - open/close 9234 (time 1764941291.06 total 40.01 last 230.35) - open/close 10000 (time 1764941294.33 total 43.29 last 233.77) - open/close 12064 (time 1764941304.33 total 53.29 last 206.37) - open/close 14266 (time 1764941314.33 total 63.29 last 220.20) - open/close 16838 (time 1764941324.34 total 73.29 last 257.11) - open/close 19406 (time 1764941334.34 total 83.29 last 256.74) - open/close 20000 (time 1764941336.40 total 85.36 last 287.76) - open/close 23325 (time 1764941346.41 total 95.36 last 332.42) - open/close 26640 (time 1764941356.41 total 105.37 last 331.29) - open/close 29546 (time 1764941366.41 total 115.37 last 290.56) - open/close 32066 (time 1764941376.42 total 125.37 last 251.96) - open/close 34139 (time 1764941386.42 total 135.37 last 207.30) - open/close 36286 (time 1764941396.42 total 145.37 last 214.68) - open/close 39126 (time 1764941406.42 total 155.37 last 283.92) - open/close 40000 (time 1764941409.36 total 158.32 last 297.06) - open/close 42738 (time 1764941419.36 total 168.32 last 273.80) - open/close 45335 (time 1764941429.36 total 178.32 last 259.69) - open/close 48297 (time 1764941439.37 total 188.32 last 296.11) - open/close 50000 (time 1764941445.48 total 194.43 last 278.62) - open/close 53313 (time 1764941455.48 total 204.43 last 331.30) - open/close 56244 (time 1764941465.48 total 214.43 last 293.04) - open/close 59069 (time 1764941475.48 total 224.44 last 282.43) - open/close 60000 (time 1764941478.51 total 227.46 last 307.39) - open/close 62621 (time 1764941488.51 total 237.47 last 262.01) total: 64768 open/close in 244.90 seconds: 264.47 ops/second - /unlink 5988 (time 1764941507.41 total 10.00 last 598.78) - /unlink 10000 (time 1764941515.53 total 18.12 last 493.89) - /unlink 15538 (time 1764941525.53 total 28.12 last 553.77) - /unlink 20000 (time 1764941533.92 total 36.51 last 531.83) - /unlink 25912 (time 1764941543.92 total 46.51 last 591.15) - /unlink 30000 (time 1764941549.86 total 52.46 last 687.78) - /unlink 33706 (time 1764941559.87 total 62.46 last 370.55) - /unlink 35049 (time 1764941569.88 total 72.47 last 134.16) - /unlink 36488 (time 1764941579.88 total 82.47 last 143.89) - /unlink 37635 (time 1764941589.88 total 92.48 last 114.63) - /unlink 38898 (time 1764941599.89 total 102.48 last 126.26) - /unlink 40000 (time 1764941609.09 total 111.68 last 119.76) - /unlink 41280 (time 1764941619.09 total 121.69 last 127.95) - /unlink 42766 (time 1764941629.10 total 131.69 last 148.51) - /unlink 44282 (time 1764941639.10 total 141.69 last 151.58) - /unlink 46520 (time 1764941649.10 total 151.70 last 223.76) - /unlink 48641 (time 1764941659.10 total 161.70 last 212.02) - /unlink 50000 (time 1764941667.13 total 169.72 last 169.41) - /unlink 51610 (time 1764941677.13 total 179.72 last 160.96) - /unlink 53211 (time 1764941687.13 total 189.73 last 160.01) - /unlink 54960 (time 1764941697.14 total 199.73 last 174.85) - /unlink 56762 (time 1764941707.14 total 209.73 last 180.16) - /unlink 58645 (time 1764941717.14 total 219.74 last 188.26) - /unlink 60000 (time 1764941723.63 total 226.22 last 208.91) - /unlink 62518 (time 1764941733.63 total 236.23 last 251.61) - /unlink 64094 (time 1764941743.64 total 246.24 last 157.52) total: 64768 /unlink in 250.27 seconds: 258.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg238-server: LNET ready to unload oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 PASS 106 (4450s) Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 oleg238-server: oleg238-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4630 sec ========== 08:37:18 (1764941838)