== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:23:44 (1761121424) Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f) oleg136-server: oleg136-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg136-server: detected 4 online CPUs by sysfs oleg136-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg136-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg136-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3891 (time 1761121474.97 total 10.00 last 389.05) - open/close 7467 (time 1761121484.97 total 20.00 last 357.56) - open/close 10000 (time 1761121493.06 total 28.09 last 313.21) - open/close 13785 (time 1761121503.06 total 38.09 last 378.47) - open/close 17519 (time 1761121513.06 total 48.09 last 373.34) - open/close 20000 (time 1761121520.46 total 55.49 last 335.29) - open/close 23186 (time 1761121530.47 total 65.49 last 318.53) - open/close 26131 (time 1761121540.47 total 75.50 last 294.46) - open/close 28732 (time 1761121550.47 total 85.50 last 260.09) - open/close 30000 (time 1761121555.68 total 90.71 last 243.03) - open/close 32556 (time 1761121565.69 total 100.72 last 255.54) - open/close 34804 (time 1761121575.69 total 110.72 last 224.76) - open/close 37092 (time 1761121585.69 total 120.72 last 228.76) - open/close 39261 (time 1761121595.70 total 130.72 last 216.79) - open/close 40000 (time 1761121601.00 total 136.03 last 139.37) - open/close 41366 (time 1761121611.01 total 146.04 last 136.48) - open/close 42665 (time 1761121621.01 total 156.04 last 129.82) - open/close 44697 (time 1761121631.01 total 166.04 last 203.19) - open/close 46846 (time 1761121641.01 total 176.04 last 214.88) - open/close 48624 (time 1761121651.02 total 186.05 last 177.75) - open/close 50000 (time 1761121657.57 total 192.60 last 209.85) - open/close 52455 (time 1761121667.58 total 202.60 last 245.47) - open/close 55032 (time 1761121677.58 total 212.61 last 257.66) - open/close 57166 (time 1761121687.58 total 222.61 last 213.33) - open/close 59444 (time 1761121697.58 total 232.61 last 227.79) - open/close 60000 (time 1761121699.64 total 234.67 last 269.76) - open/close 62747 (time 1761121709.64 total 244.67 last 274.69) total: 64768 open/close in 252.89 seconds: 256.12 ops/second - /unlink 4425 (time 1761121729.76 total 10.00 last 442.46) - /unlink 8640 (time 1761121739.76 total 20.00 last 421.46) - /unlink 10000 (time 1761121742.58 total 22.83 last 481.61) - /unlink 13659 (time 1761121752.58 total 32.83 last 365.87) - /unlink 17525 (time 1761121762.58 total 42.83 last 386.57) - /unlink 20000 (time 1761121767.87 total 48.11 last 468.38) - /unlink 24878 (time 1761121777.87 total 58.11 last 487.79) - /unlink 29652 (time 1761121787.87 total 68.11 last 477.30) - /unlink 33572 (time 1761121797.87 total 78.11 last 391.97) - /unlink 37625 (time 1761121807.87 total 88.12 last 405.23) - /unlink 40000 (time 1761121814.36 total 94.61 last 365.96) - /unlink 44162 (time 1761121824.36 total 104.61 last 416.18) - /unlink 48067 (time 1761121834.36 total 114.61 last 390.41) - /unlink 50000 (time 1761121839.11 total 119.35 last 407.75) - /unlink 54182 (time 1761121849.11 total 129.35 last 418.06) - /unlink 58039 (time 1761121859.11 total 139.36 last 385.60) - /unlink 60000 (time 1761121863.51 total 143.75 last 445.78) total: 64768 /unlink in 153.71 seconds: 421.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2726 (time 1761121903.37 total 10.00 last 272.51) - open/close 5339 (time 1761121913.38 total 20.00 last 261.26) - open/close 7971 (time 1761121923.38 total 30.01 last 263.12) - open/close 10000 (time 1761121932.53 total 39.16 last 221.79) - open/close 12728 (time 1761121942.53 total 49.16 last 272.76) - open/close 15222 (time 1761121952.53 total 59.16 last 249.32) - open/close 17615 (time 1761121962.53 total 69.16 last 239.24) - open/close 20000 (time 1761121971.88 total 78.51 last 255.25) - open/close 22629 (time 1761121981.88 total 88.51 last 262.80) - open/close 25098 (time 1761121991.88 total 98.51 last 246.82) - open/close 27405 (time 1761122001.89 total 108.52 last 230.65) - open/close 30000 (time 1761122011.15 total 117.78 last 280.11) - open/close 32792 (time 1761122021.15 total 127.78 last 279.15) - open/close 33908 (time 1761122031.15 total 137.78 last 111.58) - open/close 34713 (time 1761122041.16 total 147.79 last 80.47) - open/close 35616 (time 1761122051.17 total 157.80 last 90.22) - open/close 36309 (time 1761122061.18 total 167.80 last 69.24) - open/close 37028 (time 1761122071.18 total 177.81 last 71.88) - open/close 37747 (time 1761122081.19 total 187.82 last 71.80) - open/close 38363 (time 1761122091.20 total 197.83 last 61.57) - open/close 39029 (time 1761122101.20 total 207.83 last 66.58) - open/close 39915 (time 1761122111.20 total 217.83 last 88.56) - open/close 40852 (time 1761122121.22 total 227.85 last 93.57) - open/close 41826 (time 1761122131.22 total 237.85 last 97.38) - open/close 42822 (time 1761122141.22 total 247.85 last 99.57) - open/close 43742 (time 1761122151.24 total 257.87 last 91.87) - open/close 44647 (time 1761122161.24 total 267.87 last 90.48) - open/close 45572 (time 1761122171.24 total 277.87 last 92.47) - open/close 46502 (time 1761122181.25 total 287.88 last 92.96) - open/close 47500 (time 1761122191.25 total 297.88 last 99.73) - open/close 48415 (time 1761122201.26 total 307.89 last 91.47) - open/close 49485 (time 1761122211.26 total 317.89 last 106.94) - open/close 50000 (time 1761122215.09 total 321.72 last 134.63) - open/close 50936 (time 1761122225.11 total 331.74 last 93.43) - open/close 51819 (time 1761122235.11 total 341.74 last 88.26) - open/close 52865 (time 1761122245.12 total 351.75 last 104.53) - open/close 53734 (time 1761122255.13 total 361.76 last 86.83) - open/close 54664 (time 1761122265.13 total 371.76 last 92.96) - open/close 55704 (time 1761122275.14 total 381.77 last 103.85) - open/close 56752 (time 1761122285.15 total 391.78 last 104.73) - open/close 57862 (time 1761122295.15 total 401.78 last 110.99) - open/close 58770 (time 1761122305.15 total 411.78 last 90.78) - open/close 59646 (time 1761122315.16 total 421.79 last 87.55) - open/close 60000 (time 1761122319.18 total 425.81 last 88.12) - open/close 60915 (time 1761122329.19 total 435.81 last 91.42) - open/close 61571 (time 1761122339.20 total 445.83 last 65.49) - open/close 62090 (time 1761122349.21 total 455.84 last 51.87) - open/close 62584 (time 1761122359.21 total 465.84 last 49.38) - open/close 63072 (time 1761122369.22 total 475.85 last 48.76) - open/close 63629 (time 1761122379.23 total 485.86 last 55.64) - open/close 64067 (time 1761122389.32 total 495.95 last 43.43) - open/close 64638 (time 1761122399.32 total 505.95 last 57.09) total: 64768 open/close in 507.86 seconds: 127.53 ops/second - /unlink 1189 (time 1761122415.71 total 10.00 last 118.86) - /unlink 2385 (time 1761122425.72 total 20.01 last 119.53) - /unlink 3610 (time 1761122435.72 total 30.01 last 122.44) - /unlink 4890 (time 1761122445.73 total 40.02 last 127.92) - /unlink 6144 (time 1761122455.73 total 50.03 last 125.32) - /unlink 7510 (time 1761122465.74 total 60.03 last 136.56) - /unlink 8652 (time 1761122475.74 total 70.03 last 114.16) - /unlink 9700 (time 1761122485.75 total 80.04 last 104.71) - /unlink 10000 (time 1761122488.83 total 83.12 last 97.54) - /unlink 11016 (time 1761122498.83 total 93.13 last 101.51) - /unlink 12099 (time 1761122508.84 total 103.13 last 108.28) - /unlink 13181 (time 1761122518.84 total 113.14 last 108.11) - /unlink 14341 (time 1761122528.85 total 123.14 last 115.95) - /unlink 15411 (time 1761122538.85 total 133.14 last 106.96) - /unlink 16568 (time 1761122548.86 total 143.15 last 115.65) - /unlink 17697 (time 1761122558.86 total 153.15 last 112.82) - /unlink 18938 (time 1761122568.87 total 163.16 last 124.02) - /unlink 20000 (time 1761122576.04 total 170.34 last 148.03) - /unlink 21264 (time 1761122586.04 total 180.34 last 126.40) - /unlink 22531 (time 1761122596.05 total 190.34 last 126.63) - /unlink 23746 (time 1761122606.06 total 200.35 last 121.43) - /unlink 25081 (time 1761122616.07 total 210.36 last 133.35) - /unlink 26320 (time 1761122626.07 total 220.36 last 123.86) - /unlink 27415 (time 1761122636.08 total 230.37 last 109.38) - /unlink 28503 (time 1761122646.09 total 240.38 last 108.73) - /unlink 29800 (time 1761122656.09 total 250.38 last 129.64) - /unlink 31128 (time 1761122666.09 total 260.39 last 132.77) - /unlink 32318 (time 1761122676.10 total 270.39 last 118.96) - /unlink 33460 (time 1761122686.10 total 280.40 last 114.13) - /unlink 34623 (time 1761122696.11 total 290.40 last 116.27) - /unlink 35809 (time 1761122706.11 total 300.40 last 118.54) - /unlink 37072 (time 1761122716.12 total 310.41 last 126.22) - /unlink 38227 (time 1761122726.13 total 320.42 last 115.33) - /unlink 39307 (time 1761122736.13 total 330.42 last 108.00) - /unlink 40000 (time 1761122742.37 total 336.67 last 111.01) - /unlink 41211 (time 1761122752.38 total 346.67 last 121.05) - /unlink 42439 (time 1761122762.40 total 356.69 last 122.59) - /unlink 43691 (time 1761122772.40 total 366.69 last 125.16) - /unlink 44954 (time 1761122782.42 total 376.71 last 126.08) - /unlink 46203 (time 1761122792.42 total 386.71 last 124.87) - /unlink 47447 (time 1761122802.42 total 396.72 last 124.34) - /unlink 48741 (time 1761122812.43 total 406.72 last 129.35) - /unlink 50000 (time 1761122822.41 total 416.70 last 126.18) - /unlink 51271 (time 1761122832.41 total 426.70 last 127.07) - /unlink 52523 (time 1761122842.41 total 436.70 last 125.18) - /unlink 53738 (time 1761122852.41 total 446.70 last 121.47) - /unlink 54912 (time 1761122862.41 total 456.71 last 117.37) - /unlink 56052 (time 1761122872.42 total 466.71 last 113.96) - /unlink 57252 (time 1761122882.42 total 476.71 last 119.95) - /unlink 58533 (time 1761122892.43 total 486.72 last 128.03) - /unlink 59786 (time 1761122902.43 total 496.72 last 125.26) - /unlink 60982 (time 1761122912.43 total 506.72 last 119.57) - /unlink 62209 (time 1761122922.44 total 516.73 last 122.61) - /unlink 63476 (time 1761122932.45 total 526.74 last 126.62) total: 64768 /unlink in 536.12 seconds: 120.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 768 (time 1761122975.05 total 10.00 last 76.78) - open/close 1662 (time 1761122985.06 total 20.01 last 89.33) - open/close 2402 (time 1761122995.07 total 30.02 last 73.96) - open/close 3144 (time 1761123005.07 total 40.02 last 74.15) - open/close 3887 (time 1761123015.08 total 50.03 last 74.25) - open/close 4698 (time 1761123025.09 total 60.04 last 81.01) - open/close 5522 (time 1761123035.11 total 70.06 last 82.26) - open/close 6302 (time 1761123045.11 total 80.06 last 77.97) - open/close 7076 (time 1761123055.12 total 90.07 last 77.35) - open/close 7829 (time 1761123065.13 total 100.08 last 75.19) - open/close 8566 (time 1761123075.15 total 110.10 last 73.60) - open/close 9406 (time 1761123085.15 total 120.10 last 84.00) - open/close 10000 (time 1761123092.44 total 127.39 last 81.48) - open/close 10955 (time 1761123102.44 total 137.40 last 95.43) - open/close 11779 (time 1761123112.45 total 147.40 last 82.32) - open/close 12697 (time 1761123122.46 total 157.41 last 91.77) - open/close 13611 (time 1761123132.46 total 167.41 last 91.39) - open/close 14489 (time 1761123142.47 total 177.42 last 87.73) - open/close 15331 (time 1761123152.50 total 187.45 last 83.94) - open/close 16220 (time 1761123162.50 total 197.45 last 88.89) - open/close 17108 (time 1761123172.50 total 207.45 last 88.78) - open/close 17971 (time 1761123182.51 total 217.46 last 86.20) - open/close 18809 (time 1761123192.51 total 227.47 last 83.78) - open/close 19659 (time 1761123202.52 total 237.47 last 84.98) - open/close 20000 (time 1761123206.43 total 241.38 last 87.21) - open/close 20889 (time 1761123216.43 total 251.38 last 88.87) - open/close 21779 (time 1761123226.43 total 261.39 last 88.97) - open/close 22556 (time 1761123236.44 total 271.40 last 77.62) - open/close 23402 (time 1761123246.45 total 281.40 last 84.59) - open/close 24305 (time 1761123256.46 total 291.41 last 90.21) - open/close 25180 (time 1761123266.46 total 301.41 last 87.45) - open/close 26085 (time 1761123276.46 total 311.42 last 90.48) - open/close 26921 (time 1761123286.47 total 321.42 last 83.59) - open/close 27849 (time 1761123296.48 total 331.43 last 92.70) - open/close 28781 (time 1761123306.49 total 341.44 last 93.10) - open/close 29754 (time 1761123316.49 total 351.44 last 97.27) - open/close 30964 (time 1761123326.50 total 361.45 last 120.94) - open/close 31868 (time 1761123336.50 total 371.45 last 90.39) - open/close 32905 (time 1761123346.51 total 381.46 last 103.59) - open/close 33920 (time 1761123356.51 total 391.46 last 101.46) - open/close 34929 (time 1761123366.51 total 401.46 last 100.89) - open/close 36002 (time 1761123376.51 total 411.47 last 107.27) - open/close 37029 (time 1761123386.52 total 421.47 last 102.62) - open/close 38018 (time 1761123396.53 total 431.48 last 98.82) - open/close 39109 (time 1761123406.54 total 441.49 last 109.01) - open/close 40000 (time 1761123415.82 total 450.77 last 96.01) - open/close 40860 (time 1761123425.83 total 460.78 last 85.94) - open/close 41747 (time 1761123435.83 total 470.78 last 88.68) - open/close 42768 (time 1761123445.83 total 480.78 last 102.08) - open/close 43854 (time 1761123455.84 total 490.79 last 108.55) - open/close 44941 (time 1761123465.84 total 500.79 last 108.67) - open/close 45976 (time 1761123475.84 total 510.79 last 103.48) - open/close 46879 (time 1761123485.85 total 520.80 last 90.20) - open/close 47903 (time 1761123495.85 total 530.81 last 102.37) - open/close 48776 (time 1761123505.87 total 540.82 last 87.21) - open/close 49744 (time 1761123515.88 total 550.83 last 96.64) - open/close 50000 (time 1761123518.54 total 553.49 last 96.21) - open/close 50901 (time 1761123528.55 total 563.50 last 90.05) - open/close 51863 (time 1761123538.55 total 573.50 last 96.16) - open/close 52976 (time 1761123548.56 total 583.51 last 111.26) - open/close 54051 (time 1761123558.56 total 593.51 last 107.49) - open/close 55448 (time 1761123568.56 total 603.51 last 139.66) - open/close 56713 (time 1761123578.56 total 613.51 last 126.48) - open/close 57851 (time 1761123588.56 total 623.52 last 113.76) - open/close 58953 (time 1761123598.56 total 633.52 last 110.20) - open/close 60000 (time 1761123605.48 total 640.43 last 151.43) - open/close 61137 (time 1761123615.49 total 650.44 last 113.61) - open/close 62087 (time 1761123625.49 total 660.44 last 94.97) - open/close 63239 (time 1761123635.49 total 670.45 last 115.15) - open/close 64444 (time 1761123645.50 total 680.45 last 120.42) total: 64768 open/close in 683.16 seconds: 94.81 ops/second - /unlink 2283 (time 1761123660.31 total 10.00 last 228.22) - /unlink 4810 (time 1761123670.31 total 20.01 last 252.65) - /unlink 7453 (time 1761123680.31 total 30.01 last 264.28) - /unlink 9916 (time 1761123690.31 total 40.01 last 246.29) - /unlink 12210 (time 1761123700.32 total 50.01 last 229.30) - /unlink 14471 (time 1761123710.32 total 60.01 last 226.05) - /unlink 17140 (time 1761123720.33 total 70.02 last 266.72) - /unlink 19703 (time 1761123730.33 total 80.03 last 256.19) - /unlink 21805 (time 1761123740.33 total 90.03 last 210.16) - /unlink 23928 (time 1761123750.34 total 100.03 last 212.25) - /unlink 26162 (time 1761123760.34 total 110.03 last 223.34) - /unlink 28881 (time 1761123770.34 total 120.03 last 271.84) - /unlink 30000 (time 1761123775.20 total 124.89 last 230.53) - /unlink 31270 (time 1761123785.20 total 134.89 last 126.98) - /unlink 32245 (time 1761123795.22 total 144.91 last 97.31) - /unlink 33164 (time 1761123805.22 total 154.91 last 91.86) - /unlink 34094 (time 1761123815.23 total 164.93 last 92.89) - /unlink 35011 (time 1761123825.24 total 174.93 last 91.67) - /unlink 35962 (time 1761123835.24 total 184.94 last 95.02) - /unlink 36779 (time 1761123845.25 total 194.94 last 81.67) - /unlink 37941 (time 1761123855.25 total 204.94 last 116.18) - /unlink 39171 (time 1761123865.25 total 214.95 last 122.96) - /unlink 40000 (time 1761123872.12 total 221.81 last 120.80) - /unlink 41167 (time 1761123882.12 total 231.82 last 116.59) - /unlink 42255 (time 1761123892.13 total 241.82 last 108.74) - /unlink 43605 (time 1761123902.15 total 251.85 last 134.70) - /unlink 45090 (time 1761123912.16 total 261.85 last 148.46) - /unlink 46445 (time 1761123922.16 total 271.85 last 135.46) - /unlink 47827 (time 1761123932.16 total 281.85 last 138.16) - /unlink 49083 (time 1761123942.17 total 291.86 last 125.52) - /unlink 50000 (time 1761123949.65 total 299.34 last 122.59) - /unlink 51366 (time 1761123959.65 total 309.34 last 136.56) - /unlink 52807 (time 1761123969.65 total 319.34 last 144.09) - /unlink 54057 (time 1761123979.65 total 329.35 last 124.98) - /unlink 55300 (time 1761123989.66 total 339.35 last 124.23) - /unlink 56492 (time 1761123999.67 total 349.36 last 119.07) - /unlink 57732 (time 1761124009.68 total 359.37 last 123.84) - /unlink 59013 (time 1761124019.68 total 369.38 last 128.06) - /unlink 60000 (time 1761124028.30 total 377.99 last 114.54) - /unlink 61047 (time 1761124038.31 total 388.00 last 104.60) - /unlink 62197 (time 1761124048.31 total 398.00 last 114.99) - /unlink 63351 (time 1761124058.33 total 408.02 last 115.23) - /unlink 64524 (time 1761124068.33 total 418.03 last 117.23) total: 64768 /unlink in 420.29 seconds: 154.10 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 701 (time 1761124107.00 total 10.00 last 70.07) - open/close 1306 (time 1761124117.00 total 20.00 last 60.50) - open/close 2006 (time 1761124127.00 total 30.01 last 69.98) - open/close 2725 (time 1761124137.00 total 40.01 last 71.89) - open/close 3477 (time 1761124147.00 total 50.01 last 75.18) - open/close 4177 (time 1761124157.01 total 60.02 last 69.94) - open/close 4830 (time 1761124167.01 total 70.02 last 65.29) - open/close 5429 (time 1761124177.03 total 80.03 last 59.82) - open/close 6082 (time 1761124187.04 total 90.05 last 65.20) - open/close 6752 (time 1761124197.05 total 100.05 last 66.96) - open/close 7439 (time 1761124207.06 total 110.07 last 68.58) - open/close 8126 (time 1761124217.08 total 120.09 last 68.57) - open/close 8784 (time 1761124227.10 total 130.11 last 65.67) - open/close 9463 (time 1761124237.11 total 140.12 last 67.83) - open/close 10000 (time 1761124245.86 total 148.86 last 61.42) - open/close 10671 (time 1761124255.86 total 158.87 last 67.09) - open/close 11348 (time 1761124265.87 total 168.88 last 67.64) - open/close 12060 (time 1761124275.88 total 178.89 last 71.07) - open/close 12766 (time 1761124285.90 total 188.91 last 70.49) - open/close 13443 (time 1761124295.90 total 198.91 last 67.67) - open/close 14175 (time 1761124305.91 total 208.92 last 73.18) - open/close 14852 (time 1761124315.91 total 218.92 last 67.67) - open/close 15531 (time 1761124325.92 total 228.93 last 67.86) - open/close 16196 (time 1761124335.93 total 238.93 last 66.45) - open/close 16841 (time 1761124345.95 total 248.96 last 64.33) - open/close 17529 (time 1761124355.95 total 258.96 last 68.79) - open/close 18160 (time 1761124365.96 total 268.97 last 63.08) - open/close 18822 (time 1761124375.97 total 278.98 last 66.13) - open/close 19554 (time 1761124385.99 total 288.99 last 73.07) - open/close 20000 (time 1761124391.96 total 294.97 last 74.63) - open/close 20740 (time 1761124401.97 total 304.98 last 73.95) - open/close 21566 (time 1761124411.97 total 314.98 last 82.57) - open/close 22363 (time 1761124421.97 total 324.98 last 79.69) - open/close 23144 (time 1761124431.98 total 334.98 last 78.08) - open/close 23852 (time 1761124441.98 total 344.99 last 70.75) - open/close 24654 (time 1761124451.98 total 354.99 last 80.20) - open/close 25494 (time 1761124461.99 total 365.00 last 83.94) - open/close 26343 (time 1761124471.99 total 375.00 last 84.88) - open/close 27102 (time 1761124481.99 total 385.00 last 75.90) - open/close 27825 (time 1761124491.99 total 395.00 last 72.29) - open/close 28529 (time 1761124502.00 total 405.01 last 70.36) - open/close 29217 (time 1761124512.01 total 415.02 last 68.74) - open/close 29952 (time 1761124522.01 total 425.02 last 73.45) - open/close 30660 (time 1761124532.03 total 435.04 last 70.69) - open/close 31363 (time 1761124542.04 total 445.05 last 70.21) - open/close 32032 (time 1761124552.05 total 455.05 last 66.87) - open/close 32644 (time 1761124562.09 total 465.10 last 60.90) - open/close 33289 (time 1761124572.10 total 475.11 last 64.48) - open/close 33926 (time 1761124582.10 total 485.11 last 63.68) - open/close 34573 (time 1761124592.11 total 495.12 last 64.66) - open/close 35242 (time 1761124602.14 total 505.15 last 66.71) - open/close 35934 (time 1761124612.15 total 515.15 last 69.14) - open/close 36633 (time 1761124622.16 total 525.17 last 69.81) - open/close 37371 (time 1761124632.16 total 535.17 last 73.78) - open/close 38005 (time 1761124642.17 total 545.18 last 63.35) - open/close 38701 (time 1761124652.20 total 555.21 last 69.40) - open/close 39371 (time 1761124662.20 total 565.21 last 66.99) - open/close 40000 (time 1761124671.36 total 574.37 last 68.69) - open/close 40760 (time 1761124681.36 total 584.37 last 75.99) - open/close 41433 (time 1761124691.37 total 594.38 last 67.20) - open/close 42125 (time 1761124701.37 total 604.38 last 69.18) - open/close 42837 (time 1761124711.38 total 614.39 last 71.14) - open/close 43590 (time 1761124721.40 total 624.41 last 75.20) - open/close 44393 (time 1761124731.41 total 634.42 last 80.21) - open/close 45118 (time 1761124741.42 total 644.43 last 72.42) - open/close 45935 (time 1761124751.42 total 654.43 last 81.67) - open/close 46691 (time 1761124761.44 total 664.45 last 75.45) - open/close 47395 (time 1761124771.45 total 674.45 last 70.39) - open/close 48158 (time 1761124781.45 total 684.46 last 76.29) - open/close 48898 (time 1761124791.46 total 694.46 last 73.94) - open/close 49736 (time 1761124801.47 total 704.47 last 83.72) - open/close 50000 (time 1761124805.04 total 708.05 last 73.92) - open/close 50712 (time 1761124815.05 total 718.06 last 71.07) - open/close 51515 (time 1761124825.06 total 728.07 last 80.25) - open/close 52319 (time 1761124835.06 total 738.07 last 80.39) - open/close 53133 (time 1761124845.06 total 748.07 last 81.39) - open/close 53870 (time 1761124855.08 total 758.09 last 73.60) - open/close 54603 (time 1761124865.08 total 768.09 last 73.27) - open/close 55341 (time 1761124875.08 total 778.09 last 73.78) - open/close 56101 (time 1761124885.08 total 788.09 last 75.99) - open/close 56972 (time 1761124895.09 total 798.10 last 87.05) - open/close 57732 (time 1761124905.10 total 808.11 last 75.89) - open/close 58524 (time 1761124915.11 total 818.12 last 79.17) - open/close 59274 (time 1761124925.12 total 828.13 last 74.91) - open/close 60000 (time 1761124934.66 total 837.67 last 76.12) - open/close 60772 (time 1761124944.66 total 847.67 last 77.20) - open/close 61657 (time 1761124954.66 total 857.67 last 88.49) - open/close 62568 (time 1761124964.67 total 867.68 last 91.02) - open/close 63339 (time 1761124974.67 total 877.68 last 77.05) - open/close 64188 (time 1761124984.68 total 887.69 last 84.84) total: 64768 open/close in 894.05 seconds: 72.44 ops/second - /unlink 1623 (time 1761125004.10 total 10.00 last 162.26) - /unlink 3215 (time 1761125014.11 total 20.01 last 159.01) - /unlink 4730 (time 1761125024.11 total 30.02 last 151.46) - /unlink 6035 (time 1761125034.12 total 40.02 last 130.47) - /unlink 7579 (time 1761125044.12 total 50.02 last 154.37) - /unlink 9458 (time 1761125054.12 total 60.02 last 187.87) - /unlink 10000 (time 1761125057.40 total 63.31 last 164.99) - /unlink 11466 (time 1761125067.41 total 73.31 last 146.53) - /unlink 12841 (time 1761125077.41 total 83.31 last 137.48) - /unlink 14345 (time 1761125087.41 total 93.32 last 150.36) - /unlink 16004 (time 1761125097.42 total 103.32 last 165.82) - /unlink 17631 (time 1761125107.42 total 113.33 last 162.59) - /unlink 19049 (time 1761125117.43 total 123.34 last 141.68) - /unlink 20000 (time 1761125124.77 total 130.67 last 129.67) - /unlink 21271 (time 1761125134.77 total 140.67 last 127.08) - /unlink 22536 (time 1761125144.77 total 150.68 last 126.46) - /unlink 24002 (time 1761125154.77 total 160.68 last 146.57) - /unlink 25409 (time 1761125164.79 total 170.69 last 140.49) - /unlink 26759 (time 1761125174.80 total 180.70 last 134.90) - /unlink 28062 (time 1761125184.80 total 190.70 last 130.27) - /unlink 29667 (time 1761125194.80 total 200.71 last 160.44) - /unlink 30000 (time 1761125197.20 total 203.10 last 138.91) - /unlink 31403 (time 1761125207.22 total 213.13 last 139.97) - /unlink 32885 (time 1761125217.23 total 223.13 last 148.18) - /unlink 34379 (time 1761125227.23 total 233.13 last 149.39) - /unlink 36064 (time 1761125237.23 total 243.13 last 168.48) - /unlink 37676 (time 1761125247.23 total 253.13 last 161.17) - /unlink 39256 (time 1761125257.24 total 263.14 last 157.87) - /unlink 40000 (time 1761125262.02 total 267.92 last 155.68) - /unlink 41566 (time 1761125272.02 total 277.92 last 156.59) - /unlink 43165 (time 1761125282.02 total 287.92 last 159.87) - /unlink 44763 (time 1761125292.02 total 297.93 last 159.75) - /unlink 46310 (time 1761125302.02 total 307.93 last 154.66) - /unlink 47910 (time 1761125312.03 total 317.93 last 159.96) - /unlink 49573 (time 1761125322.03 total 327.93 last 166.25) - /unlink 50000 (time 1761125324.20 total 330.11 last 196.66) - /unlink 51723 (time 1761125334.20 total 340.11 last 172.26) - /unlink 52840 (time 1761125344.21 total 350.11 last 111.64) - /unlink 53628 (time 1761125354.21 total 360.12 last 78.78) - /unlink 54232 (time 1761125364.21 total 370.12 last 60.38) - /unlink 54836 (time 1761125374.22 total 380.12 last 60.36) - /unlink 55710 (time 1761125384.23 total 390.13 last 87.32) - /unlink 56569 (time 1761125394.24 total 400.14 last 85.82) - /unlink 57410 (time 1761125404.24 total 410.15 last 84.08) - /unlink 58319 (time 1761125414.29 total 420.19 last 90.48) - /unlink 58977 (time 1761125424.29 total 430.20 last 65.77) - /unlink 59797 (time 1761125434.30 total 440.20 last 81.96) - /unlink 60795 (time 1761125444.30 total 450.20 last 99.80) - /unlink 61840 (time 1761125454.30 total 460.20 last 104.50) - /unlink 62893 (time 1761125464.30 total 470.21 last 105.25) - /unlink 63987 (time 1761125474.33 total 480.23 last 109.12) total: 64768 /unlink in 487.73 seconds: 132.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 675 (time 1761125515.52 total 10.00 last 67.47) - open/close 1329 (time 1761125525.53 total 20.01 last 65.36) - open/close 2019 (time 1761125535.54 total 30.03 last 68.88) - open/close 2628 (time 1761125545.54 total 40.03 last 60.90) - open/close 3227 (time 1761125555.55 total 50.03 last 59.88) - open/close 3801 (time 1761125565.56 total 60.04 last 57.34) - open/close 4428 (time 1761125575.57 total 70.05 last 62.65) - open/close 5025 (time 1761125585.58 total 80.06 last 59.64) - open/close 5687 (time 1761125595.59 total 90.07 last 66.13) - open/close 6388 (time 1761125605.59 total 100.08 last 70.05) - open/close 7041 (time 1761125615.60 total 110.09 last 65.24) - open/close 7672 (time 1761125625.60 total 120.09 last 63.09) - open/close 8313 (time 1761125635.62 total 130.10 last 64.01) - open/close 8968 (time 1761125645.64 total 140.12 last 65.38) - open/close 9569 (time 1761125655.65 total 150.13 last 60.05) - open/close 10000 (time 1761125662.44 total 156.92 last 63.43) - open/close 10617 (time 1761125672.45 total 166.93 last 61.66) - open/close 11237 (time 1761125682.45 total 176.94 last 61.96) - open/close 11736 (time 1761125692.46 total 186.94 last 49.89) - open/close 12392 (time 1761125702.46 total 196.94 last 65.57) - open/close 13016 (time 1761125712.47 total 206.95 last 62.35) - open/close 13619 (time 1761125722.49 total 216.97 last 60.19) - open/close 14240 (time 1761125732.50 total 226.98 last 62.00) - open/close 14852 (time 1761125742.51 total 236.99 last 61.15) - open/close 15419 (time 1761125752.52 total 247.00 last 56.65) - open/close 15870 (time 1761125762.54 total 257.02 last 45.02) - open/close 16406 (time 1761125772.55 total 267.03 last 53.53) - open/close 16958 (time 1761125782.56 total 277.04 last 55.15) - open/close 17433 (time 1761125792.56 total 287.04 last 47.49) - open/close 17845 (time 1761125802.57 total 297.06 last 41.15) - open/close 18214 (time 1761125812.59 total 307.08 last 36.82) - open/close 18710 (time 1761125822.59 total 317.08 last 49.59) - open/close 19267 (time 1761125832.61 total 327.09 last 55.64) - open/close 19881 (time 1761125842.61 total 337.09 last 61.37) - open/close 20430 (time 1761125852.64 total 347.13 last 54.72) - open/close 20939 (time 1761125862.66 total 357.14 last 50.83) - open/close 21374 (time 1761125872.67 total 367.15 last 43.46) - open/close 21976 (time 1761125882.67 total 377.16 last 60.16) - open/close 22537 (time 1761125892.68 total 387.16 last 56.05) - open/close 23126 (time 1761125902.69 total 397.17 last 58.86) - open/close 23721 (time 1761125912.70 total 407.19 last 59.41) - open/close 24362 (time 1761125922.72 total 417.20 last 64.00) - open/close 24952 (time 1761125932.73 total 427.21 last 58.96) - open/close 25572 (time 1761125942.74 total 437.22 last 61.93) - open/close 26219 (time 1761125952.77 total 447.26 last 64.45) - open/close 26855 (time 1761125962.78 total 457.26 last 63.58) - open/close 27493 (time 1761125972.79 total 467.28 last 63.69) - open/close 28120 (time 1761125982.82 total 477.30 last 62.57) - open/close 28779 (time 1761125992.82 total 487.30 last 65.89) - open/close 29402 (time 1761126002.82 total 497.30 last 62.29) - open/close 30000 (time 1761126012.75 total 507.23 last 60.23) - open/close 30548 (time 1761126022.75 total 517.24 last 54.77) - open/close 31201 (time 1761126032.77 total 527.25 last 65.21) - open/close 31887 (time 1761126042.78 total 537.26 last 68.51) - open/close 32507 (time 1761126052.78 total 547.26 last 61.99) - open/close 33125 (time 1761126062.79 total 557.27 last 61.76) - open/close 33763 (time 1761126072.79 total 567.27 last 63.80) - open/close 34387 (time 1761126082.80 total 577.28 last 62.35) - open/close 35026 (time 1761126092.81 total 587.29 last 63.83) - open/close 35668 (time 1761126102.82 total 597.30 last 64.13) - open/close 36313 (time 1761126112.82 total 607.30 last 64.49) - open/close 36977 (time 1761126122.83 total 617.31 last 66.34) - open/close 37607 (time 1761126132.84 total 627.32 last 62.94) - open/close 38238 (time 1761126142.86 total 637.34 last 62.96) - open/close 38859 (time 1761126152.87 total 647.35 last 62.06) - open/close 39507 (time 1761126162.88 total 657.36 last 64.71) - open/close 40000 (time 1761126171.29 total 665.77 last 58.62) - open/close 40504 (time 1761126181.30 total 675.78 last 50.36) - open/close 41093 (time 1761126191.30 total 685.78 last 58.88) - open/close 41719 (time 1761126201.31 total 695.79 last 62.54) - open/close 42312 (time 1761126211.32 total 705.80 last 59.26) - open/close 42928 (time 1761126221.33 total 715.81 last 61.52) - open/close 43525 (time 1761126231.33 total 725.82 last 59.68) - open/close 44139 (time 1761126241.34 total 735.82 last 61.38) - open/close 44693 (time 1761126251.34 total 745.82 last 55.38) - open/close 45269 (time 1761126261.36 total 755.84 last 57.50) - open/close 45853 (time 1761126271.36 total 765.85 last 58.37) - open/close 46465 (time 1761126281.38 total 775.86 last 61.13) - open/close 47067 (time 1761126291.39 total 785.88 last 60.09) - open/close 47618 (time 1761126301.39 total 795.88 last 55.10) - open/close 48163 (time 1761126311.42 total 805.90 last 54.37) - open/close 48741 (time 1761126321.45 total 815.93 last 57.62) - open/close 49297 (time 1761126331.46 total 825.95 last 55.53) - open/close 49827 (time 1761126341.47 total 835.95 last 52.98) - open/close 50000 (time 1761126344.35 total 838.84 last 59.97) - open/close 50619 (time 1761126354.37 total 848.85 last 61.78) - open/close 51187 (time 1761126364.38 total 858.87 last 56.73) - open/close 51787 (time 1761126374.39 total 868.87 last 59.98) - open/close 52400 (time 1761126384.39 total 878.88 last 61.25) - open/close 52974 (time 1761126394.41 total 888.89 last 57.34) - open/close 53527 (time 1761126404.41 total 898.90 last 55.26) - open/close 54116 (time 1761126414.42 total 908.91 last 58.84) - open/close 54722 (time 1761126424.43 total 918.91 last 60.56) - open/close 55280 (time 1761126434.43 total 928.91 last 55.80) - open/close 55767 (time 1761126444.44 total 938.92 last 48.66) - open/close 56364 (time 1761126454.46 total 948.94 last 59.58) - open/close 56947 (time 1761126464.48 total 958.96 last 58.18) - open/close 57576 (time 1761126474.49 total 968.97 last 62.82) - open/close 58217 (time 1761126484.49 total 978.98 last 64.08) - open/close 58815 (time 1761126494.50 total 988.98 last 59.78) - open/close 59385 (time 1761126504.52 total 999.00 last 56.88) - open/close 60000 (time 1761126514.45 total 1008.93 last 61.91) - open/close 60634 (time 1761126524.46 total 1018.94 last 63.36) - open/close 61261 (time 1761126534.46 total 1028.94 last 62.69) - open/close 61822 (time 1761126544.47 total 1038.95 last 56.03) - open/close 62422 (time 1761126554.49 total 1048.97 last 59.90) - open/close 63062 (time 1761126564.50 total 1058.98 last 63.96) - open/close 63676 (time 1761126574.51 total 1069.00 last 61.29) - open/close 64268 (time 1761126584.53 total 1079.02 last 59.09) total: 64768 open/close in 1086.95 seconds: 59.59 ops/second - /unlink 1001 (time 1761126608.10 total 10.04 last 99.73) - /unlink 2000 (time 1761126618.11 total 20.04 last 99.87) - /unlink 3170 (time 1761126628.11 total 30.04 last 116.97) - /unlink 4232 (time 1761126638.11 total 40.05 last 106.13) - /unlink 4998 (time 1761126648.12 total 50.06 last 76.53) - /unlink 5972 (time 1761126658.13 total 60.06 last 97.37) - /unlink 6988 (time 1761126668.13 total 70.07 last 101.52) - /unlink 7893 (time 1761126678.14 total 80.07 last 90.46) - /unlink 8940 (time 1761126688.14 total 90.08 last 104.67) - /unlink 9937 (time 1761126698.15 total 100.09 last 99.59) - /unlink 11004 (time 1761126708.15 total 110.09 last 106.68) - /unlink 11925 (time 1761126718.16 total 120.09 last 92.08) - /unlink 12838 (time 1761126728.17 total 130.10 last 91.21) - /unlink 13838 (time 1761126738.17 total 140.10 last 99.99) - /unlink 14785 (time 1761126748.17 total 150.10 last 94.69) - /unlink 15772 (time 1761126758.17 total 160.10 last 98.69) - /unlink 16841 (time 1761126768.17 total 170.11 last 106.87) - /unlink 17910 (time 1761126778.18 total 180.11 last 106.86) - /unlink 18933 (time 1761126788.18 total 190.12 last 102.23) - /unlink 19801 (time 1761126798.20 total 200.13 last 86.68) - /unlink 20000 (time 1761126800.35 total 202.28 last 92.55) - /unlink 21056 (time 1761126810.35 total 212.29 last 105.53) - /unlink 21982 (time 1761126820.36 total 222.29 last 92.55) - /unlink 22910 (time 1761126830.36 total 232.30 last 92.78) - /unlink 23868 (time 1761126840.37 total 242.31 last 95.71) - /unlink 24835 (time 1761126850.38 total 252.32 last 96.60) - /unlink 25746 (time 1761126860.40 total 262.33 last 90.94) - /unlink 26696 (time 1761126870.41 total 272.34 last 94.89) - /unlink 27687 (time 1761126880.42 total 282.35 last 99.03) - /unlink 28711 (time 1761126890.42 total 292.35 last 102.37) - /unlink 29742 (time 1761126900.43 total 302.36 last 103.02) - /unlink 30000 (time 1761126902.69 total 304.62 last 114.18) - /unlink 30962 (time 1761126912.70 total 314.63 last 96.09) - /unlink 31923 (time 1761126922.71 total 324.64 last 96.03) - /unlink 32986 (time 1761126932.71 total 334.64 last 106.27) - /unlink 34021 (time 1761126942.71 total 344.65 last 103.43) - /unlink 35033 (time 1761126952.72 total 354.65 last 101.14) - /unlink 36043 (time 1761126962.72 total 364.65 last 101.00) - /unlink 37093 (time 1761126972.72 total 374.66 last 104.98) - /unlink 38116 (time 1761126982.73 total 384.66 last 102.24) - /unlink 39104 (time 1761126992.74 total 394.68 last 98.64) - /unlink 40000 (time 1761127001.36 total 403.29 last 104.00) - /unlink 41023 (time 1761127011.37 total 413.30 last 102.25) - /unlink 42082 (time 1761127021.37 total 423.30 last 105.85) - /unlink 43074 (time 1761127031.37 total 433.31 last 99.17) - /unlink 44017 (time 1761127041.38 total 443.31 last 94.23) - /unlink 44978 (time 1761127051.38 total 453.32 last 96.07) - /unlink 45985 (time 1761127061.38 total 463.32 last 100.68) - /unlink 47009 (time 1761127071.39 total 473.32 last 102.34) - /unlink 47987 (time 1761127081.39 total 483.33 last 97.76) - /unlink 48905 (time 1761127091.39 total 493.33 last 91.80) - /unlink 49983 (time 1761127101.40 total 503.33 last 107.77) - /unlink 51047 (time 1761127111.40 total 513.34 last 106.33) - /unlink 52016 (time 1761127121.41 total 523.34 last 96.88) - /unlink 53046 (time 1761127131.41 total 533.35 last 102.95) - /unlink 54042 (time 1761127141.42 total 543.35 last 99.53) - /unlink 55054 (time 1761127151.42 total 553.36 last 101.15) - /unlink 55914 (time 1761127161.44 total 563.38 last 85.85) - /unlink 56721 (time 1761127171.45 total 573.38 last 80.65) - /unlink 57720 (time 1761127181.47 total 583.41 last 99.66) - /unlink 58695 (time 1761127191.47 total 593.41 last 97.48) - /unlink 59633 (time 1761127201.48 total 603.41 last 93.74) - /unlink 60000 (time 1761127205.17 total 607.11 last 99.31) - /unlink 61049 (time 1761127215.18 total 617.11 last 104.88) - /unlink 62092 (time 1761127225.19 total 627.12 last 104.20) - /unlink 63051 (time 1761127235.19 total 637.13 last 95.83) - /unlink 64032 (time 1761127245.19 total 647.13 last 98.08) total: 64768 /unlink in 654.41 seconds: 98.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local oleg136-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg136-server: LNET ready to unload oleg136-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2