== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:22:57 (1761283377) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg207-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32587 331 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3193 (time 1761283425.75 total 10.00 last 319.15) - open/close 5600 (time 1761283435.75 total 20.00 last 240.70) - open/close 7940 (time 1761283445.75 total 30.01 last 233.93) - open/close 10000 (time 1761283455.18 total 39.44 last 218.45) - open/close 12880 (time 1761283465.18 total 49.44 last 287.99) - open/close 15471 (time 1761283475.19 total 59.44 last 259.02) - open/close 17546 (time 1761283485.19 total 69.44 last 207.44) - open/close 19274 (time 1761283495.19 total 79.45 last 172.76) - open/close 20000 (time 1761283499.08 total 83.33 last 186.92) - open/close 22237 (time 1761283509.08 total 93.33 last 223.69) - open/close 24256 (time 1761283519.08 total 103.33 last 201.83) - open/close 26036 (time 1761283529.08 total 113.34 last 177.96) - open/close 28104 (time 1761283539.08 total 123.34 last 206.78) - open/close 30000 (time 1761283548.91 total 133.16 last 192.98) - open/close 32139 (time 1761283558.91 total 143.16 last 213.88) - open/close 35782 (time 1761283568.91 total 153.17 last 364.20) - open/close 39537 (time 1761283578.91 total 163.17 last 375.47) - open/close 43090 (time 1761283588.92 total 173.17 last 355.11) - open/close 46134 (time 1761283598.92 total 183.17 last 304.39) - open/close 49620 (time 1761283608.92 total 193.18 last 348.49) - open/close 51923 (time 1761283618.92 total 203.18 last 230.25) - open/close 54029 (time 1761283628.93 total 213.18 last 210.50) - open/close 56001 (time 1761283638.93 total 223.19 last 197.14) - open/close 58760 (time 1761283648.93 total 233.19 last 275.87) - open/close 60000 (time 1761283653.37 total 237.63 last 279.02) - open/close 63679 (time 1761283663.38 total 247.63 last 367.90) total: 64768 open/close in 250.63 seconds: 258.42 ops/second - /unlink 6815 (time 1761283677.85 total 10.00 last 681.44) - /unlink 10000 (time 1761283682.68 total 14.84 last 658.73) - /unlink 17097 (time 1761283692.68 total 24.84 last 709.61) - /unlink 20000 (time 1761283696.99 total 29.14 last 674.12) - /unlink 27923 (time 1761283706.99 total 39.14 last 792.27) - /unlink 30000 (time 1761283709.60 total 41.75 last 795.66) - /unlink 37738 (time 1761283719.60 total 51.76 last 773.73) - /unlink 40000 (time 1761283722.48 total 54.64 last 785.40) - /unlink 47459 (time 1761283732.48 total 64.64 last 745.82) - /unlink 50000 (time 1761283735.49 total 67.64 last 845.03) - /unlink 58230 (time 1761283745.49 total 77.64 last 823.00) - /unlink 60000 (time 1761283747.92 total 80.07 last 729.79) total: 64768 /unlink in 87.02 seconds: 744.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4764 (time 1761283784.41 total 10.00 last 476.31) - open/close 9506 (time 1761283794.41 total 20.00 last 474.17) - open/close 14085 (time 1761283804.41 total 30.00 last 457.84) - open/close 18778 (time 1761283814.41 total 40.00 last 469.28) - open/close 20000 (time 1761283816.91 total 42.51 last 488.58) - open/close 24210 (time 1761283826.91 total 52.51 last 420.92) - open/close 28560 (time 1761283836.92 total 62.51 last 434.93) - open/close 30000 (time 1761283839.77 total 65.37 last 503.75) - open/close 34683 (time 1761283849.77 total 75.37 last 468.24) - open/close 39239 (time 1761283859.78 total 85.37 last 455.55) - open/close 43980 (time 1761283869.78 total 95.37 last 474.02) - open/close 48803 (time 1761283879.78 total 105.37 last 482.24) - open/close 50000 (time 1761283882.23 total 107.83 last 487.98) - open/close 55042 (time 1761283892.23 total 117.83 last 504.19) - open/close 59734 (time 1761283902.23 total 127.83 last 469.16) total: 64768 open/close in 137.65 seconds: 470.53 ops/second - /unlink 8212 (time 1761283923.40 total 10.00 last 821.17) - /unlink 10000 (time 1761283925.51 total 12.11 last 847.73) - /unlink 17502 (time 1761283935.51 total 22.11 last 750.13) - /unlink 20000 (time 1761283938.82 total 25.42 last 753.94) - /unlink 28008 (time 1761283948.82 total 35.43 last 800.67) - /unlink 30000 (time 1761283951.02 total 37.63 last 904.66) - /unlink 38577 (time 1761283961.02 total 47.63 last 857.69) - /unlink 45951 (time 1761283971.02 total 57.63 last 737.38) - /unlink 50000 (time 1761283976.52 total 63.12 last 737.11) - /unlink 57194 (time 1761283986.52 total 73.12 last 719.35) - /unlink 60000 (time 1761283990.59 total 77.19 last 689.46) total: 64768 /unlink in 83.22 seconds: 778.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4442 (time 1761284026.51 total 10.00 last 444.00) - open/close 8809 (time 1761284036.51 total 20.01 last 436.65) - open/close 10000 (time 1761284039.29 total 22.78 last 429.30) - open/close 14943 (time 1761284049.29 total 32.78 last 494.27) - open/close 20000 (time 1761284058.97 total 42.46 last 522.27) - open/close 25007 (time 1761284068.97 total 52.46 last 500.68) - open/close 29510 (time 1761284078.97 total 62.46 last 450.25) - open/close 34019 (time 1761284088.97 total 72.47 last 450.89) - open/close 38529 (time 1761284098.98 total 82.47 last 450.92) - open/close 40000 (time 1761284102.19 total 85.68 last 458.19) - open/close 44372 (time 1761284112.19 total 95.68 last 437.14) - open/close 48659 (time 1761284122.19 total 105.68 last 428.69) - open/close 50000 (time 1761284125.31 total 108.80 last 429.51) - open/close 54329 (time 1761284135.31 total 118.80 last 432.82) - open/close 58797 (time 1761284145.31 total 128.80 last 446.73) - open/close 60000 (time 1761284148.01 total 131.50 last 445.61) - open/close 64519 (time 1761284158.01 total 141.50 last 451.88) total: 64768 open/close in 142.23 seconds: 455.38 ops/second - /unlink 7100 (time 1761284170.15 total 10.00 last 709.91) - /unlink 10000 (time 1761284173.89 total 13.74 last 775.43) - /unlink 17446 (time 1761284183.89 total 23.74 last 744.59) - /unlink 20000 (time 1761284187.54 total 27.39 last 700.23) - /unlink 26969 (time 1761284197.54 total 37.39 last 696.89) - /unlink 30000 (time 1761284200.88 total 40.73 last 906.71) - /unlink 38157 (time 1761284210.88 total 50.73 last 815.62) - /unlink 40000 (time 1761284213.20 total 53.05 last 796.48) - /unlink 48469 (time 1761284223.21 total 63.06 last 845.52) - /unlink 50000 (time 1761284225.32 total 65.17 last 728.15) - /unlink 58383 (time 1761284235.32 total 75.17 last 838.25) total: 64768 /unlink in 82.17 seconds: 788.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4627 (time 1761284286.16 total 10.00 last 462.67) - open/close 9347 (time 1761284296.16 total 20.00 last 471.92) - open/close 12515 (time 1761284306.17 total 30.01 last 316.58) - open/close 14461 (time 1761284316.17 total 40.01 last 194.53) - open/close 17104 (time 1761284326.17 total 50.02 last 264.21) - open/close 20000 (time 1761284333.89 total 57.74 last 375.03) - open/close 24434 (time 1761284343.90 total 67.74 last 443.27) - open/close 28963 (time 1761284353.90 total 77.74 last 452.82) - open/close 30000 (time 1761284356.21 total 80.05 last 448.95) - open/close 34401 (time 1761284366.21 total 90.05 last 440.07) - open/close 38854 (time 1761284376.21 total 100.06 last 445.22) - open/close 40000 (time 1761284379.16 total 103.01 last 388.42) - open/close 43914 (time 1761284389.16 total 113.01 last 391.37) - open/close 47435 (time 1761284399.16 total 123.01 last 352.07) - open/close 50000 (time 1761284409.01 total 132.85 last 260.55) - open/close 53068 (time 1761284419.01 total 142.85 last 306.72) - open/close 56494 (time 1761284429.01 total 152.86 last 342.53) - open/close 59623 (time 1761284439.02 total 162.86 last 312.79) - open/close 63410 (time 1761284449.02 total 172.86 last 378.68) total: 64768 open/close in 175.82 seconds: 368.37 ops/second - /unlink 7518 (time 1761284463.30 total 10.00 last 751.51) - /unlink 10000 (time 1761284467.10 total 13.80 last 653.21) - /unlink 18231 (time 1761284477.10 total 23.81 last 822.96) - /unlink 20000 (time 1761284479.78 total 26.48 last 661.25) - /unlink 27152 (time 1761284489.78 total 36.48 last 715.15) - /unlink 30000 (time 1761284494.01 total 40.72 last 672.51) - /unlink 37893 (time 1761284504.01 total 50.72 last 789.29) - /unlink 40000 (time 1761284506.47 total 53.17 last 858.79) - /unlink 46103 (time 1761284516.48 total 63.18 last 609.37) - /unlink 47595 (time 1761284526.48 total 73.18 last 149.20) - /unlink 49248 (time 1761284536.48 total 83.19 last 165.28) - /unlink 50000 (time 1761284541.02 total 87.72 last 165.89) - /unlink 51588 (time 1761284551.02 total 97.72 last 158.71) - /unlink 52876 (time 1761284561.02 total 107.73 last 128.79) - /unlink 54006 (time 1761284571.03 total 117.73 last 112.94) - /unlink 55135 (time 1761284581.04 total 127.74 last 112.78) - /unlink 56374 (time 1761284591.04 total 137.74 last 123.88) - /unlink 57975 (time 1761284601.04 total 147.75 last 160.04) - /unlink 59802 (time 1761284611.05 total 157.75 last 182.67) - /unlink 61926 (time 1761284621.05 total 167.75 last 212.34) - /unlink 64025 (time 1761284631.05 total 177.76 last 209.78) total: 64768 /unlink in 182.50 seconds: 354.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 912 (time 1761284663.60 total 10.02 last 91.05) - open/close 1860 (time 1761284673.61 total 20.02 last 94.72) - open/close 2803 (time 1761284683.61 total 30.03 last 94.23) - open/close 3791 (time 1761284693.62 total 40.04 last 98.75) - open/close 4960 (time 1761284703.62 total 50.04 last 116.89) - open/close 5852 (time 1761284713.62 total 60.04 last 89.16) - open/close 6677 (time 1761284723.63 total 70.04 last 82.48) - open/close 7598 (time 1761284733.64 total 80.05 last 92.01) - open/close 8214 (time 1761284743.64 total 90.06 last 61.55) - open/close 8619 (time 1761284753.65 total 100.07 last 40.48) - open/close 8964 (time 1761284763.66 total 110.07 last 34.48) - open/close 9345 (time 1761284773.67 total 120.08 last 38.06) - open/close 9895 (time 1761284783.68 total 130.09 last 54.95) - open/close 10000 (time 1761284785.81 total 132.23 last 49.15) - open/close 10535 (time 1761284795.82 total 142.23 last 53.49) - open/close 11271 (time 1761284805.84 total 152.26 last 73.43) - open/close 12006 (time 1761284815.85 total 162.27 last 73.43) - open/close 12654 (time 1761284825.85 total 172.27 last 64.76) - open/close 13397 (time 1761284835.88 total 182.30 last 74.09) - open/close 14141 (time 1761284845.90 total 192.32 last 74.28) - open/close 14838 (time 1761284855.91 total 202.33 last 69.63) - open/close 15603 (time 1761284865.91 total 212.33 last 76.49) - open/close 16414 (time 1761284875.91 total 222.33 last 81.09) - open/close 17137 (time 1761284885.91 total 232.33 last 72.30) - open/close 17885 (time 1761284895.91 total 242.33 last 74.80) - open/close 18585 (time 1761284905.92 total 252.34 last 69.97) - open/close 19223 (time 1761284915.96 total 262.38 last 63.54) - open/close 19957 (time 1761284925.97 total 272.38 last 73.35) - open/close 20698 (time 1761284935.97 total 282.39 last 74.05) - open/close 21443 (time 1761284945.99 total 292.40 last 74.40) - open/close 22169 (time 1761284955.99 total 302.41 last 72.58) - open/close 22914 (time 1761284966.00 total 312.42 last 74.41) - open/close 23654 (time 1761284976.01 total 322.42 last 73.95) - open/close 24378 (time 1761284986.01 total 332.42 last 72.40) - open/close 25133 (time 1761284996.02 total 342.44 last 75.38) - open/close 25888 (time 1761285006.03 total 352.45 last 75.41) - open/close 26639 (time 1761285016.04 total 362.46 last 75.07) - open/close 27350 (time 1761285026.06 total 372.48 last 70.95) - open/close 28142 (time 1761285036.07 total 382.49 last 79.12) - open/close 28860 (time 1761285046.07 total 392.49 last 71.80) - open/close 29597 (time 1761285056.07 total 402.49 last 73.67) - open/close 30000 (time 1761285061.35 total 407.77 last 76.38) - open/close 30633 (time 1761285071.37 total 417.79 last 63.19) - open/close 31345 (time 1761285081.39 total 427.80 last 71.07) - open/close 32081 (time 1761285091.39 total 437.81 last 73.55) - open/close 32890 (time 1761285101.40 total 447.82 last 80.84) - open/close 33630 (time 1761285111.40 total 457.82 last 73.97) - open/close 34399 (time 1761285121.41 total 467.83 last 76.86) - open/close 35140 (time 1761285131.41 total 477.83 last 74.09) - open/close 35862 (time 1761285141.41 total 487.83 last 72.17) - open/close 36448 (time 1761285151.42 total 497.84 last 58.56) - open/close 37026 (time 1761285161.43 total 507.85 last 57.72) - open/close 37791 (time 1761285171.44 total 517.86 last 76.42) - open/close 38545 (time 1761285181.45 total 527.87 last 75.34) - open/close 39247 (time 1761285191.46 total 537.88 last 70.12) - open/close 39974 (time 1761285201.47 total 547.89 last 72.65) - open/close 40643 (time 1761285211.47 total 557.89 last 66.89) - open/close 41341 (time 1761285221.48 total 567.90 last 69.73) - open/close 42055 (time 1761285231.49 total 577.91 last 71.34) - open/close 42747 (time 1761285241.50 total 587.92 last 69.15) - open/close 43476 (time 1761285251.50 total 597.92 last 72.88) - open/close 44201 (time 1761285261.51 total 607.93 last 72.40) - open/close 44846 (time 1761285271.52 total 617.94 last 64.48) - open/close 45548 (time 1761285281.52 total 627.94 last 70.18) - open/close 46273 (time 1761285291.53 total 637.95 last 72.42) - open/close 46974 (time 1761285301.53 total 647.95 last 70.09) - open/close 47717 (time 1761285311.53 total 657.95 last 74.27) - open/close 48446 (time 1761285321.55 total 667.97 last 72.80) - open/close 49110 (time 1761285331.56 total 677.97 last 66.34) - open/close 49828 (time 1761285341.57 total 687.98 last 71.73) - open/close 50000 (time 1761285344.03 total 690.45 last 69.87) - open/close 50690 (time 1761285354.04 total 700.46 last 68.89) - open/close 51387 (time 1761285364.05 total 710.47 last 69.62) - open/close 52069 (time 1761285374.06 total 720.48 last 68.15) - open/close 52738 (time 1761285384.07 total 730.49 last 66.87) - open/close 53420 (time 1761285394.08 total 740.50 last 68.13) - open/close 54076 (time 1761285404.09 total 750.50 last 65.54) - open/close 54728 (time 1761285414.10 total 760.51 last 65.13) - open/close 55406 (time 1761285424.10 total 770.52 last 67.75) - open/close 56064 (time 1761285434.11 total 780.53 last 65.75) - open/close 56685 (time 1761285444.14 total 790.56 last 61.93) - open/close 57219 (time 1761285454.14 total 800.56 last 53.40) - open/close 57837 (time 1761285464.14 total 810.56 last 61.79) - open/close 58493 (time 1761285474.15 total 820.57 last 65.54) - open/close 59176 (time 1761285484.17 total 830.59 last 68.16) - open/close 59865 (time 1761285494.17 total 840.59 last 68.90) - open/close 60590 (time 1761285504.18 total 850.59 last 72.48) - open/close 61302 (time 1761285514.18 total 860.60 last 71.14) - open/close 62010 (time 1761285524.19 total 870.61 last 70.73) - open/close 62749 (time 1761285534.20 total 880.62 last 73.85) - open/close 63463 (time 1761285544.21 total 890.63 last 71.34) - open/close 64155 (time 1761285554.22 total 900.64 last 69.10) total: 64768 open/close in 909.44 seconds: 71.22 ops/second - /unlink 724 (time 1761285576.58 total 10.01 last 72.34) - /unlink 1420 (time 1761285586.59 total 20.02 last 69.52) - /unlink 2190 (time 1761285596.60 total 30.03 last 76.93) - /unlink 2884 (time 1761285606.61 total 40.04 last 69.33) - /unlink 3635 (time 1761285616.62 total 50.05 last 75.01) - /unlink 4322 (time 1761285626.64 total 60.06 last 68.60) - /unlink 5025 (time 1761285636.64 total 70.07 last 70.29) - /unlink 5744 (time 1761285646.64 total 80.07 last 71.85) - /unlink 6495 (time 1761285656.64 total 90.07 last 75.10) - /unlink 7161 (time 1761285666.65 total 100.08 last 66.59) - /unlink 7889 (time 1761285676.66 total 110.09 last 72.66) - /unlink 8640 (time 1761285686.67 total 120.10 last 75.06) - /unlink 9367 (time 1761285696.68 total 130.11 last 72.63) - /unlink 10000 (time 1761285706.55 total 139.98 last 64.11) - /unlink 10739 (time 1761285716.55 total 149.98 last 73.90) - /unlink 11439 (time 1761285726.56 total 159.99 last 69.95) - /unlink 12165 (time 1761285736.57 total 170.00 last 72.55) - /unlink 12891 (time 1761285746.57 total 180.00 last 72.55) - /unlink 13645 (time 1761285756.58 total 190.01 last 75.36) - /unlink 14274 (time 1761285766.59 total 200.02 last 62.86) - /unlink 14962 (time 1761285776.59 total 210.02 last 68.78) - /unlink 15675 (time 1761285786.60 total 220.03 last 71.19) - /unlink 16372 (time 1761285796.61 total 230.04 last 69.66) - /unlink 17064 (time 1761285806.62 total 240.05 last 69.16) - /unlink 17732 (time 1761285816.63 total 250.06 last 66.72) - /unlink 18444 (time 1761285826.64 total 260.07 last 71.09) - /unlink 19143 (time 1761285836.65 total 270.08 last 69.88) - /unlink 19811 (time 1761285846.65 total 280.08 last 66.78) - /unlink 20000 (time 1761285849.93 total 283.35 last 57.70) - /unlink 20633 (time 1761285859.93 total 293.36 last 63.25) - /unlink 21225 (time 1761285869.93 total 303.36 last 59.19) - /unlink 21911 (time 1761285879.94 total 313.37 last 68.57) - /unlink 22619 (time 1761285889.94 total 323.37 last 70.78) - /unlink 23317 (time 1761285899.95 total 333.38 last 69.75) - /unlink 24050 (time 1761285909.95 total 343.38 last 73.29) - /unlink 24753 (time 1761285919.95 total 353.38 last 70.28) - /unlink 25466 (time 1761285929.96 total 363.39 last 71.23) - /unlink 26192 (time 1761285939.96 total 373.39 last 72.59) - /unlink 26876 (time 1761285949.98 total 383.41 last 68.31) - /unlink 27547 (time 1761285959.98 total 393.41 last 67.09) - /unlink 28306 (time 1761285969.98 total 403.41 last 75.87) - /unlink 28961 (time 1761285979.99 total 413.42 last 65.45) - /unlink 29664 (time 1761285990.00 total 423.43 last 70.20) - /unlink 30000 (time 1761285995.81 total 429.24 last 57.84) - /unlink 30761 (time 1761286005.82 total 439.25 last 76.07) - /unlink 31497 (time 1761286015.82 total 449.25 last 73.57) - /unlink 32222 (time 1761286025.83 total 459.26 last 72.41) - /unlink 33006 (time 1761286035.84 total 469.26 last 78.39) - /unlink 33814 (time 1761286045.84 total 479.27 last 80.77) - /unlink 34595 (time 1761286055.84 total 489.27 last 78.09) - /unlink 35404 (time 1761286065.85 total 499.28 last 80.86) - /unlink 36221 (time 1761286075.85 total 509.28 last 81.67) - /unlink 37042 (time 1761286085.85 total 519.28 last 82.09) - /unlink 37842 (time 1761286095.86 total 529.29 last 79.95) - /unlink 38589 (time 1761286105.88 total 539.31 last 74.55) - /unlink 39341 (time 1761286115.88 total 549.31 last 75.18) - /unlink 40000 (time 1761286123.70 total 557.13 last 84.27) - /unlink 40813 (time 1761286133.70 total 567.13 last 81.29) - /unlink 41602 (time 1761286143.71 total 577.14 last 78.86) - /unlink 42361 (time 1761286153.72 total 587.15 last 75.83) - /unlink 43144 (time 1761286163.72 total 597.15 last 78.24) - /unlink 43949 (time 1761286173.73 total 607.16 last 80.43) - /unlink 44775 (time 1761286183.74 total 617.17 last 82.54) - /unlink 45642 (time 1761286193.75 total 627.18 last 86.63) - /unlink 46516 (time 1761286203.76 total 637.19 last 87.32) - /unlink 47390 (time 1761286213.77 total 647.20 last 87.29) - /unlink 48244 (time 1761286223.77 total 657.20 last 85.37) - /unlink 49083 (time 1761286233.78 total 667.21 last 83.89) - /unlink 49907 (time 1761286243.78 total 677.21 last 82.37) - /unlink 50731 (time 1761286253.79 total 687.22 last 82.34) - /unlink 51581 (time 1761286263.79 total 697.22 last 84.97) - /unlink 52389 (time 1761286273.79 total 707.22 last 80.76) - /unlink 53141 (time 1761286283.80 total 717.23 last 75.19) - /unlink 53958 (time 1761286293.81 total 727.24 last 81.59) - /unlink 54810 (time 1761286303.81 total 737.24 last 85.20) - /unlink 55608 (time 1761286313.81 total 747.24 last 79.78) - /unlink 56431 (time 1761286323.87 total 757.30 last 81.82) - /unlink 57189 (time 1761286333.87 total 767.30 last 75.78) - /unlink 57980 (time 1761286343.88 total 777.31 last 79.05) - /unlink 58809 (time 1761286353.88 total 787.31 last 82.90) - /unlink 59630 (time 1761286363.89 total 797.32 last 82.04) - /unlink 60000 (time 1761286368.67 total 802.10 last 77.32) - /unlink 60809 (time 1761286378.67 total 812.10 last 80.90) - /unlink 61590 (time 1761286388.68 total 822.11 last 78.06) - /unlink 62392 (time 1761286398.68 total 832.11 last 80.19) - /unlink 63215 (time 1761286408.69 total 842.12 last 82.25) - /unlink 64019 (time 1761286418.69 total 852.12 last 80.40) total: 64768 /unlink in 861.60 seconds: 75.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2