== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:00:22 (1746140422) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) oleg216-server: oleg216-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: MODOPTS_LIBCFS= oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg216-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1102 (time 1746140489.90 total 10.00 last 110.16) - open/close 2257 (time 1746140499.91 total 20.01 last 115.43) - open/close 3446 (time 1746140509.91 total 30.01 last 118.88) - open/close 4558 (time 1746140519.91 total 40.01 last 111.20) - open/close 5662 (time 1746140529.92 total 50.02 last 110.35) - open/close 6935 (time 1746140539.92 total 60.02 last 127.28) - open/close 8210 (time 1746140549.92 total 70.02 last 127.48) - open/close 9488 (time 1746140559.92 total 80.02 last 127.79) - open/close 10000 (time 1746140563.82 total 83.92 last 131.21) - open/close 11553 (time 1746140573.83 total 93.93 last 155.12) - open/close 12942 (time 1746140583.84 total 103.94 last 138.87) - open/close 14099 (time 1746140593.84 total 113.94 last 115.67) - open/close 15352 (time 1746140603.84 total 123.94 last 125.27) - open/close 16504 (time 1746140613.85 total 133.95 last 115.15) - open/close 17791 (time 1746140623.85 total 143.95 last 128.66) - open/close 19250 (time 1746140633.85 total 153.96 last 145.81) - open/close 20000 (time 1746140638.85 total 158.95 last 150.27) - open/close 21267 (time 1746140648.86 total 168.96 last 126.46) - open/close 22521 (time 1746140658.87 total 178.97 last 125.32) - open/close 23897 (time 1746140668.87 total 188.97 last 137.55) - open/close 25277 (time 1746140678.88 total 198.98 last 137.91) - open/close 26584 (time 1746140688.88 total 208.98 last 130.68) - open/close 28113 (time 1746140698.89 total 218.99 last 152.78) - open/close 29433 (time 1746140708.89 total 228.99 last 131.97) - open/close 30000 (time 1746140713.06 total 233.16 last 135.97) - open/close 31132 (time 1746140723.07 total 243.17 last 113.16) - open/close 32588 (time 1746140733.07 total 253.17 last 145.56) - open/close 34438 (time 1746140743.08 total 263.18 last 184.84) - open/close 35927 (time 1746140753.08 total 273.18 last 148.83) - open/close 37305 (time 1746140763.08 total 283.18 last 137.79) - open/close 38659 (time 1746140773.09 total 293.19 last 135.36) - open/close 40000 (time 1746140782.33 total 302.43 last 145.00) - open/close 41414 (time 1746140792.33 total 312.43 last 141.39) - open/close 42981 (time 1746140802.34 total 322.44 last 156.64) - open/close 44438 (time 1746140812.34 total 332.44 last 145.65) - open/close 45871 (time 1746140822.35 total 342.45 last 143.25) - open/close 47265 (time 1746140832.35 total 352.45 last 139.39) - open/close 48529 (time 1746140842.35 total 362.45 last 126.34) - open/close 49977 (time 1746140852.35 total 372.45 last 144.77) - open/close 51356 (time 1746140862.35 total 382.45 last 137.90) - open/close 52859 (time 1746140872.36 total 392.46 last 150.24) - open/close 54206 (time 1746140882.36 total 402.46 last 134.68) - open/close 55465 (time 1746140892.36 total 412.46 last 125.83) - open/close 56660 (time 1746140902.37 total 422.47 last 119.46) - open/close 57948 (time 1746140912.37 total 432.47 last 128.79) - open/close 59270 (time 1746140922.37 total 442.47 last 132.15) - open/close 60000 (time 1746140927.71 total 447.81 last 136.86) - open/close 61421 (time 1746140937.71 total 457.81 last 142.05) - open/close 62771 (time 1746140947.72 total 467.82 last 134.93) - open/close 64522 (time 1746140957.72 total 477.82 last 175.09) total: 64768 open/close in 479.03 seconds: 135.21 ops/second - /unlink 2647 (time 1746140970.14 total 10.00 last 264.70) - /unlink 5175 (time 1746140980.15 total 20.01 last 252.61) - /unlink 7480 (time 1746140990.15 total 30.01 last 230.47) - /unlink 9574 (time 1746141000.15 total 40.01 last 209.38) - /unlink 11662 (time 1746141010.16 total 50.01 last 208.73) - /unlink 13762 (time 1746141020.16 total 60.02 last 209.96) - /unlink 15947 (time 1746141030.16 total 70.02 last 218.48) - /unlink 17976 (time 1746141040.16 total 80.02 last 202.79) - /unlink 20000 (time 1746141048.86 total 88.72 last 232.64) - /unlink 22359 (time 1746141058.87 total 98.72 last 235.83) - /unlink 24714 (time 1746141068.87 total 108.73 last 235.37) - /unlink 27170 (time 1746141078.87 total 118.73 last 245.58) - /unlink 29592 (time 1746141088.88 total 128.73 last 242.16) - /unlink 32035 (time 1746141098.88 total 138.73 last 244.24) - /unlink 34558 (time 1746141108.88 total 148.73 last 252.30) - /unlink 37139 (time 1746141118.88 total 158.74 last 258.07) - /unlink 39424 (time 1746141128.88 total 168.74 last 228.40) - /unlink 40000 (time 1746141131.48 total 171.34 last 221.79) - /unlink 42091 (time 1746141141.48 total 181.34 last 209.08) - /unlink 44238 (time 1746141151.48 total 191.34 last 214.67) - /unlink 46468 (time 1746141161.48 total 201.34 last 223.00) - /unlink 48403 (time 1746141171.49 total 211.34 last 193.45) - /unlink 50000 (time 1746141179.46 total 219.32 last 200.18) - /unlink 52209 (time 1746141189.47 total 229.32 last 220.84) - /unlink 54484 (time 1746141199.47 total 239.32 last 227.46) - /unlink 56066 (time 1746141209.47 total 249.33 last 158.15) - /unlink 57957 (time 1746141219.47 total 259.33 last 189.06) - /unlink 59972 (time 1746141229.48 total 269.33 last 201.41) - /unlink 61993 (time 1746141239.48 total 279.34 last 201.99) - /unlink 64215 (time 1746141249.49 total 289.35 last 222.06) total: 64768 /unlink in 292.08 seconds: 221.75 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1234 (time 1746141279.05 total 10.00 last 123.37) - open/close 2730 (time 1746141289.06 total 20.01 last 149.56) - open/close 4260 (time 1746141299.06 total 30.01 last 152.93) - open/close 5718 (time 1746141309.06 total 40.01 last 145.79) - open/close 7173 (time 1746141319.06 total 50.01 last 145.47) - open/close 8730 (time 1746141329.07 total 60.02 last 155.61) - open/close 10000 (time 1746141337.28 total 68.23 last 154.70) - open/close 11503 (time 1746141347.29 total 78.23 last 150.21) - open/close 13127 (time 1746141357.29 total 88.24 last 162.35) - open/close 15123 (time 1746141367.29 total 98.24 last 199.57) - open/close 16654 (time 1746141377.29 total 108.24 last 153.07) - open/close 18161 (time 1746141387.29 total 118.24 last 150.67) - open/close 19686 (time 1746141397.30 total 128.25 last 152.44) - open/close 20000 (time 1746141399.41 total 130.36 last 148.64) - open/close 21355 (time 1746141409.41 total 140.36 last 135.49) - open/close 22781 (time 1746141419.41 total 150.36 last 142.59) - open/close 24385 (time 1746141429.42 total 160.37 last 160.32) - open/close 26033 (time 1746141439.42 total 170.37 last 164.78) - open/close 28445 (time 1746141449.47 total 180.42 last 240.04) - open/close 30000 (time 1746141456.71 total 187.66 last 214.73) - open/close 31714 (time 1746141466.71 total 197.66 last 171.38) - open/close 33370 (time 1746141476.71 total 207.66 last 165.53) - open/close 35153 (time 1746141486.71 total 217.66 last 178.30) - open/close 37004 (time 1746141496.72 total 227.67 last 185.00) - open/close 38481 (time 1746141506.72 total 237.67 last 147.68) - open/close 40000 (time 1746141515.57 total 246.51 last 171.73) - open/close 41647 (time 1746141525.57 total 256.52 last 164.58) - open/close 43875 (time 1746141535.57 total 266.52 last 222.78) - open/close 45818 (time 1746141545.58 total 276.53 last 194.24) - open/close 47595 (time 1746141555.58 total 286.53 last 177.65) - open/close 49127 (time 1746141565.58 total 296.53 last 153.15) - open/close 50000 (time 1746141571.32 total 302.27 last 152.11) - open/close 51601 (time 1746141581.34 total 312.28 last 159.89) - open/close 53201 (time 1746141591.34 total 322.29 last 159.98) - open/close 54797 (time 1746141601.34 total 332.29 last 159.57) - open/close 56681 (time 1746141611.34 total 342.29 last 188.35) - open/close 58659 (time 1746141621.34 total 352.29 last 197.79) - open/close 60000 (time 1746141628.30 total 359.25 last 192.63) - open/close 61708 (time 1746141638.31 total 369.26 last 170.75) - open/close 63353 (time 1746141648.31 total 379.26 last 164.47) total: 64768 open/close in 389.14 seconds: 166.44 ops/second - /unlink 2911 (time 1746141669.77 total 10.00 last 291.06) - /unlink 5593 (time 1746141679.77 total 20.00 last 268.18) - /unlink 8386 (time 1746141689.77 total 30.00 last 279.23) - /unlink 10000 (time 1746141695.63 total 35.86 last 275.72) - /unlink 12613 (time 1746141705.63 total 45.86 last 261.17) - /unlink 15528 (time 1746141715.63 total 55.86 last 291.49) - /unlink 17921 (time 1746141725.63 total 65.86 last 239.28) - /unlink 20000 (time 1746141733.18 total 73.41 last 275.61) - /unlink 22214 (time 1746141743.18 total 83.41 last 221.34) - /unlink 24771 (time 1746141753.18 total 93.41 last 255.61) - /unlink 27260 (time 1746141763.18 total 103.42 last 248.87) - /unlink 29623 (time 1746141773.19 total 113.42 last 236.24) - /unlink 31781 (time 1746141783.19 total 123.42 last 215.71) - /unlink 33771 (time 1746141793.19 total 133.42 last 198.98) - /unlink 35928 (time 1746141803.19 total 143.42 last 215.65) - /unlink 38088 (time 1746141813.20 total 153.43 last 215.90) - /unlink 39992 (time 1746141823.21 total 163.44 last 190.21) - /unlink 41851 (time 1746141833.21 total 173.44 last 185.84) - /unlink 44122 (time 1746141843.21 total 183.45 last 227.04) - /unlink 46040 (time 1746141853.22 total 193.45 last 191.78) - /unlink 47984 (time 1746141863.22 total 203.45 last 194.40) - /unlink 50000 (time 1746141872.67 total 212.90 last 213.18) - /unlink 52217 (time 1746141882.67 total 222.90 last 221.69) - /unlink 54406 (time 1746141892.68 total 232.91 last 218.84) - /unlink 57122 (time 1746141902.68 total 242.91 last 271.56) - /unlink 59791 (time 1746141912.68 total 252.91 last 266.83) - /unlink 63113 (time 1746141922.68 total 262.91 last 332.17) total: 64768 /unlink in 268.67 seconds: 241.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1317 (time 1746141960.52 total 10.04 last 131.12) - open/close 2767 (time 1746141970.53 total 20.05 last 144.85) - open/close 4184 (time 1746141980.53 total 30.06 last 141.62) - open/close 5640 (time 1746141990.54 total 40.06 last 145.55) - open/close 7178 (time 1746142000.54 total 50.07 last 153.76) - open/close 8477 (time 1746142010.54 total 60.07 last 129.83) - open/close 9761 (time 1746142020.55 total 70.08 last 128.27) - open/close 11274 (time 1746142030.55 total 80.08 last 151.30) - open/close 12908 (time 1746142040.56 total 90.09 last 163.26) - open/close 14257 (time 1746142050.56 total 100.09 last 134.90) - open/close 15796 (time 1746142060.57 total 110.10 last 153.80) - open/close 17276 (time 1746142070.57 total 120.10 last 147.97) - open/close 19068 (time 1746142080.58 total 130.11 last 179.08) - open/close 20000 (time 1746142085.48 total 135.00 last 190.28) - open/close 21489 (time 1746142095.48 total 145.01 last 148.81) - open/close 22836 (time 1746142105.49 total 155.01 last 134.64) - open/close 24086 (time 1746142115.49 total 165.02 last 124.94) - open/close 25518 (time 1746142125.49 total 175.02 last 143.17) - open/close 27435 (time 1746142135.50 total 185.02 last 191.62) - open/close 29080 (time 1746142145.50 total 195.03 last 164.38) - open/close 30000 (time 1746142150.67 total 200.20 last 177.97) - open/close 31574 (time 1746142160.68 total 210.20 last 157.33) - open/close 33089 (time 1746142170.69 total 220.21 last 151.38) - open/close 34577 (time 1746142180.69 total 230.22 last 148.76) - open/close 36047 (time 1746142190.69 total 240.22 last 146.99) - open/close 37543 (time 1746142200.69 total 250.22 last 149.59) - open/close 38922 (time 1746142210.69 total 260.22 last 137.87) - open/close 40000 (time 1746142217.59 total 267.11 last 156.37) - open/close 41357 (time 1746142227.59 total 277.12 last 135.60) - open/close 42816 (time 1746142237.59 total 287.12 last 145.90) - open/close 44116 (time 1746142247.60 total 297.13 last 129.89) - open/close 45550 (time 1746142257.61 total 307.14 last 143.30) - open/close 46900 (time 1746142267.61 total 317.14 last 134.95) - open/close 48085 (time 1746142277.62 total 327.14 last 118.46) - open/close 49238 (time 1746142287.62 total 337.15 last 115.26) - open/close 50000 (time 1746142294.66 total 344.19 last 108.18) - open/close 51115 (time 1746142304.67 total 354.20 last 111.41) - open/close 52182 (time 1746142314.67 total 364.20 last 106.68) - open/close 53344 (time 1746142324.67 total 374.20 last 116.18) - open/close 54778 (time 1746142334.68 total 384.21 last 143.31) - open/close 56090 (time 1746142344.68 total 394.21 last 131.15) - open/close 57471 (time 1746142354.69 total 404.22 last 137.97) - open/close 58851 (time 1746142364.70 total 414.22 last 137.96) - open/close 60000 (time 1746142373.13 total 422.66 last 136.24) - open/close 61269 (time 1746142383.14 total 432.67 last 126.81) - open/close 62767 (time 1746142393.14 total 442.67 last 149.78) - open/close 64191 (time 1746142403.15 total 452.68 last 142.26) total: 64768 open/close in 456.58 seconds: 141.86 ops/second - /unlink 2148 (time 1746142418.63 total 10.00 last 214.80) - /unlink 4266 (time 1746142428.64 total 20.01 last 211.69) - /unlink 6435 (time 1746142438.64 total 30.01 last 216.80) - /unlink 8368 (time 1746142448.64 total 40.01 last 193.29) - /unlink 10000 (time 1746142456.12 total 47.49 last 218.18) - /unlink 12532 (time 1746142466.12 total 57.49 last 253.19) - /unlink 15173 (time 1746142476.13 total 67.49 last 264.03) - /unlink 18157 (time 1746142486.13 total 77.50 last 298.28) - /unlink 20000 (time 1746142492.74 total 84.11 last 278.88) - /unlink 22935 (time 1746142502.74 total 94.11 last 293.45) - /unlink 26113 (time 1746142512.74 total 104.11 last 317.72) - /unlink 28655 (time 1746142522.74 total 114.11 last 254.19) - /unlink 30000 (time 1746142528.58 total 119.95 last 230.30) - /unlink 32284 (time 1746142538.59 total 129.95 last 228.33) - /unlink 34591 (time 1746142548.59 total 139.96 last 230.57) - /unlink 37169 (time 1746142558.59 total 149.96 last 257.78) - /unlink 40000 (time 1746142568.45 total 159.82 last 287.14) - /unlink 42890 (time 1746142578.46 total 169.82 last 288.93) - /unlink 45898 (time 1746142588.46 total 179.82 last 300.77) - /unlink 48605 (time 1746142598.46 total 189.83 last 270.55) - /unlink 50000 (time 1746142604.40 total 195.77 last 234.84) - /unlink 51988 (time 1746142614.40 total 205.77 last 198.75) - /unlink 53809 (time 1746142624.41 total 215.77 last 182.07) - /unlink 55664 (time 1746142634.41 total 225.77 last 185.48) - /unlink 58321 (time 1746142644.41 total 235.78 last 265.67) - /unlink 60000 (time 1746142650.05 total 241.42 last 297.70) - /unlink 62628 (time 1746142660.05 total 251.42 last 262.77) total: 64768 /unlink in 260.20 seconds: 248.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1424 (time 1746142700.59 total 10.01 last 142.31) - open/close 2804 (time 1746142710.59 total 20.01 last 137.97) - open/close 4211 (time 1746142720.59 total 30.01 last 140.67) - open/close 5790 (time 1746142730.59 total 40.01 last 157.90) - open/close 7238 (time 1746142740.59 total 50.01 last 144.79) - open/close 8612 (time 1746142750.60 total 60.02 last 137.31) - open/close 10000 (time 1746142760.22 total 69.64 last 144.32) - open/close 11368 (time 1746142770.22 total 79.64 last 136.79) - open/close 12843 (time 1746142780.22 total 89.64 last 147.46) - open/close 14297 (time 1746142790.22 total 99.64 last 145.39) - open/close 15645 (time 1746142800.22 total 109.64 last 134.75) - open/close 16944 (time 1746142810.23 total 119.65 last 129.85) - open/close 18404 (time 1746142820.23 total 129.65 last 145.97) - open/close 19779 (time 1746142830.23 total 139.65 last 137.48) - open/close 21085 (time 1746142840.23 total 149.65 last 130.55) - open/close 22514 (time 1746142850.23 total 159.65 last 142.89) - open/close 24023 (time 1746142860.24 total 169.66 last 150.82) - open/close 25331 (time 1746142870.24 total 179.66 last 130.75) - open/close 26824 (time 1746142880.25 total 189.67 last 149.28) - open/close 28248 (time 1746142890.25 total 199.67 last 142.33) - open/close 29708 (time 1746142900.25 total 209.67 last 145.96) - open/close 31195 (time 1746142910.25 total 219.67 last 148.67) - open/close 32691 (time 1746142920.26 total 229.68 last 149.57) - open/close 33850 (time 1746142930.26 total 239.68 last 115.88) - open/close 35092 (time 1746142940.26 total 249.68 last 124.20) - open/close 36400 (time 1746142950.26 total 259.68 last 130.76) - open/close 37789 (time 1746142960.26 total 269.68 last 138.87) - open/close 39162 (time 1746142970.27 total 279.69 last 137.23) - open/close 40000 (time 1746142976.24 total 285.66 last 140.45) - open/close 41250 (time 1746142986.24 total 295.66 last 124.96) - open/close 42375 (time 1746142996.24 total 305.66 last 112.44) - open/close 43758 (time 1746143006.25 total 315.67 last 138.24) - open/close 45268 (time 1746143016.25 total 325.67 last 150.97) - open/close 46468 (time 1746143026.26 total 335.68 last 119.89) - open/close 47764 (time 1746143036.26 total 345.68 last 129.55) - open/close 49105 (time 1746143046.27 total 355.69 last 134.07) - open/close 50000 (time 1746143054.05 total 363.47 last 115.02) - open/close 51228 (time 1746143064.05 total 373.47 last 122.78) - open/close 52438 (time 1746143074.06 total 383.48 last 120.92) - open/close 53640 (time 1746143084.06 total 393.48 last 120.18) - open/close 54740 (time 1746143094.07 total 403.49 last 109.89) - open/close 56002 (time 1746143104.07 total 413.49 last 126.16) - open/close 57228 (time 1746143114.09 total 423.51 last 122.41) - open/close 58465 (time 1746143124.09 total 433.51 last 123.63) - open/close 59761 (time 1746143134.10 total 443.52 last 129.49) - open/close 60970 (time 1746143144.10 total 453.52 last 120.86) - open/close 62107 (time 1746143154.11 total 463.53 last 113.64) - open/close 63195 (time 1746143164.11 total 473.53 last 108.76) - open/close 64308 (time 1746143174.12 total 483.54 last 111.24) total: 64768 open/close in 487.35 seconds: 132.90 ops/second - /unlink 1860 (time 1746143190.07 total 10.00 last 185.95) - /unlink 4040 (time 1746143200.07 total 20.01 last 217.94) - /unlink 5862 (time 1746143210.08 total 30.01 last 182.05) - /unlink 7697 (time 1746143220.08 total 40.02 last 183.45) - /unlink 9644 (time 1746143230.09 total 50.02 last 194.63) - /unlink 10000 (time 1746143232.16 total 52.09 last 171.99) - /unlink 12131 (time 1746143242.16 total 62.09 last 213.08) - /unlink 14085 (time 1746143252.17 total 72.10 last 195.18) - /unlink 16022 (time 1746143262.17 total 82.11 last 193.64) - /unlink 17905 (time 1746143272.17 total 92.11 last 188.27) - /unlink 19805 (time 1746143282.18 total 102.11 last 189.99) - /unlink 21759 (time 1746143292.18 total 112.11 last 195.35) - /unlink 23763 (time 1746143302.18 total 122.12 last 200.25) - /unlink 25614 (time 1746143312.19 total 132.12 last 184.99) - /unlink 27531 (time 1746143322.20 total 142.13 last 191.54) - /unlink 29387 (time 1746143332.20 total 152.13 last 185.59) - /unlink 30000 (time 1746143336.01 total 155.94 last 160.92) - /unlink 31872 (time 1746143346.02 total 165.95 last 187.02) - /unlink 33747 (time 1746143356.02 total 175.95 last 187.47) - /unlink 35583 (time 1746143366.02 total 185.96 last 183.53) - /unlink 37472 (time 1746143376.02 total 195.96 last 188.89) - /unlink 39243 (time 1746143386.03 total 205.96 last 177.05) - /unlink 40000 (time 1746143389.98 total 209.92 last 191.31) - /unlink 41585 (time 1746143399.98 total 219.92 last 158.49) - /unlink 43409 (time 1746143409.99 total 229.92 last 182.35) - /unlink 44926 (time 1746143420.00 total 239.93 last 151.51) - /unlink 46638 (time 1746143430.00 total 249.93 last 171.16) - /unlink 48510 (time 1746143440.01 total 259.94 last 187.07) - /unlink 50000 (time 1746143447.33 total 267.26 last 203.65) - /unlink 52123 (time 1746143457.33 total 277.26 last 212.24) - /unlink 54216 (time 1746143467.33 total 287.26 last 209.29) - /unlink 56541 (time 1746143477.33 total 297.27 last 232.41) - /unlink 58458 (time 1746143487.33 total 307.27 last 191.69) - /unlink 60000 (time 1746143495.04 total 314.97 last 200.04) - /unlink 62294 (time 1746143505.05 total 324.98 last 229.32) - /unlink 64409 (time 1746143515.05 total 334.98 last 211.42) total: 64768 /unlink in 336.98 seconds: 192.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1239 (time 1746143548.12 total 10.01 last 123.81) - open/close 2511 (time 1746143558.13 total 20.01 last 127.13) - open/close 3719 (time 1746143568.13 total 30.02 last 120.74) - open/close 4909 (time 1746143578.14 total 40.03 last 118.89) - open/close 6128 (time 1746143588.15 total 50.03 last 121.83) - open/close 7280 (time 1746143598.15 total 60.03 last 115.20) - open/close 8599 (time 1746143608.16 total 70.04 last 131.83) - open/close 9903 (time 1746143618.16 total 80.05 last 130.29) - open/close 11184 (time 1746143628.17 total 90.05 last 128.05) - open/close 12497 (time 1746143638.17 total 100.05 last 131.25) - open/close 13860 (time 1746143648.18 total 110.06 last 136.19) - open/close 15124 (time 1746143658.18 total 120.06 last 126.40) - open/close 16356 (time 1746143668.18 total 130.06 last 123.18) - open/close 17545 (time 1746143678.18 total 140.07 last 118.86) - open/close 18547 (time 1746143688.19 total 150.08 last 100.09) - open/close 19654 (time 1746143698.20 total 160.08 last 110.66) - open/close 20000 (time 1746143700.98 total 162.86 last 124.56) - open/close 21234 (time 1746143710.98 total 172.86 last 123.34) - open/close 22523 (time 1746143720.98 total 182.87 last 128.85) - open/close 23701 (time 1746143731.00 total 192.88 last 117.67) - open/close 24925 (time 1746143741.00 total 202.88 last 122.38) - open/close 26241 (time 1746143751.00 total 212.88 last 131.55) - open/close 27510 (time 1746143761.01 total 222.89 last 126.79) - open/close 29007 (time 1746143771.02 total 232.90 last 149.59) - open/close 30000 (time 1746143778.11 total 239.99 last 140.06) - open/close 31345 (time 1746143788.11 total 249.99 last 134.43) - open/close 32792 (time 1746143798.11 total 260.00 last 144.69) - open/close 34294 (time 1746143808.11 total 270.00 last 150.19) - open/close 35838 (time 1746143818.12 total 280.00 last 154.31) - open/close 37445 (time 1746143828.12 total 290.01 last 160.65) - open/close 38868 (time 1746143838.13 total 300.02 last 142.12) - open/close 40000 (time 1746143846.57 total 308.45 last 134.21) - open/close 41328 (time 1746143856.57 total 318.46 last 132.76) - open/close 42697 (time 1746143866.58 total 328.46 last 136.86) - open/close 44124 (time 1746143876.58 total 338.46 last 142.67) - open/close 45716 (time 1746143886.58 total 348.46 last 159.16) - open/close 47468 (time 1746143896.58 total 358.47 last 175.13) - open/close 49208 (time 1746143906.59 total 368.47 last 173.95) - open/close 50000 (time 1746143912.11 total 374.00 last 143.29) - open/close 51377 (time 1746143922.12 total 384.01 last 137.59) - open/close 52745 (time 1746143932.12 total 394.01 last 136.79) - open/close 54095 (time 1746143942.13 total 404.01 last 134.94) - open/close 55543 (time 1746143952.13 total 414.01 last 144.77) - open/close 56959 (time 1746143962.13 total 424.01 last 141.58) - open/close 58478 (time 1746143972.14 total 434.02 last 151.84) - open/close 59817 (time 1746143982.14 total 444.03 last 133.78) - open/close 61344 (time 1746143992.15 total 454.03 last 152.62) - open/close 62827 (time 1746144002.15 total 464.04 last 148.23) - open/close 64280 (time 1746144012.16 total 474.04 last 145.22) total: 64768 open/close in 477.97 seconds: 135.51 ops/second - /unlink 2153 (time 1746144027.83 total 10.01 last 215.16) - /unlink 4156 (time 1746144037.83 total 20.01 last 200.25) - /unlink 6148 (time 1746144047.83 total 30.01 last 199.18) - /unlink 8119 (time 1746144057.84 total 40.02 last 196.95) - /unlink 9727 (time 1746144067.84 total 50.02 last 160.78) - /unlink 11604 (time 1746144077.85 total 60.02 last 187.66) - /unlink 13563 (time 1746144087.85 total 70.03 last 195.81) - /unlink 15372 (time 1746144097.86 total 80.03 last 180.80) - /unlink 17427 (time 1746144107.86 total 90.03 last 205.44) - /unlink 19507 (time 1746144117.86 total 100.04 last 207.96) - /unlink 21751 (time 1746144127.86 total 110.04 last 224.40) - /unlink 23671 (time 1746144137.86 total 120.04 last 191.96) - /unlink 25938 (time 1746144147.86 total 130.04 last 226.66) - /unlink 28263 (time 1746144157.87 total 140.04 last 232.42) - /unlink 30000 (time 1746144166.67 total 148.84 last 197.43) - /unlink 32194 (time 1746144176.67 total 158.84 last 219.36) - /unlink 34344 (time 1746144186.67 total 168.84 last 214.99) - /unlink 36615 (time 1746144196.67 total 178.85 last 227.07) - /unlink 38949 (time 1746144206.68 total 188.85 last 233.28) - /unlink 40000 (time 1746144212.49 total 194.66 last 180.86) - /unlink 42086 (time 1746144222.49 total 204.66 last 208.58) - /unlink 44228 (time 1746144232.49 total 214.67 last 214.14) - /unlink 46640 (time 1746144242.49 total 224.67 last 241.12) - /unlink 49414 (time 1746144252.50 total 234.67 last 277.35) - /unlink 50000 (time 1746144254.86 total 237.04 last 247.85) - /unlink 52164 (time 1746144264.86 total 247.04 last 216.32) - /unlink 54470 (time 1746144274.87 total 257.04 last 230.51) - /unlink 56900 (time 1746144284.88 total 267.05 last 242.81) - /unlink 59238 (time 1746144294.88 total 277.05 last 233.71) - /unlink 60000 (time 1746144298.21 total 280.39 last 228.53) - /unlink 62009 (time 1746144308.22 total 290.39 last 200.87) - /unlink 64072 (time 1746144318.22 total 300.39 last 206.24) total: 64768 /unlink in 304.26 seconds: 212.87 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server unloading modules via unload_modules_local on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2