== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:01:39 (1746136899) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg430-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1357 (time 1746136954.46 total 10.00 last 135.68) - open/close 2617 (time 1746136964.47 total 20.00 last 125.99) - open/close 3807 (time 1746136974.47 total 30.01 last 118.95) - open/close 5100 (time 1746136984.47 total 40.01 last 129.24) - open/close 6288 (time 1746136994.48 total 50.01 last 118.78) - open/close 7317 (time 1746137004.48 total 60.02 last 102.85) - open/close 8397 (time 1746137014.48 total 70.02 last 107.99) - open/close 9803 (time 1746137024.48 total 80.02 last 140.56) - open/close 11128 (time 1746137034.49 total 90.02 last 132.46) - open/close 12467 (time 1746137044.49 total 100.03 last 133.89) - open/close 13952 (time 1746137054.49 total 110.03 last 148.48) - open/close 15411 (time 1746137064.50 total 120.03 last 145.81) - open/close 16827 (time 1746137074.50 total 130.04 last 141.50) - open/close 18357 (time 1746137084.50 total 140.04 last 152.96) - open/close 19719 (time 1746137094.51 total 150.04 last 136.17) - open/close 20000 (time 1746137096.67 total 152.21 last 129.71) - open/close 21472 (time 1746137106.68 total 162.22 last 147.11) - open/close 22758 (time 1746137116.69 total 172.23 last 128.47) - open/close 24022 (time 1746137126.70 total 182.24 last 126.29) - open/close 25184 (time 1746137136.70 total 192.24 last 116.15) - open/close 26482 (time 1746137146.71 total 202.24 last 129.74) - open/close 27843 (time 1746137156.71 total 212.25 last 136.08) - open/close 29019 (time 1746137166.71 total 222.25 last 117.54) - open/close 30000 (time 1746137173.95 total 229.49 last 135.54) - open/close 31310 (time 1746137183.95 total 239.49 last 130.98) - open/close 32973 (time 1746137193.95 total 249.49 last 166.28) - open/close 34385 (time 1746137203.96 total 259.50 last 141.06) - open/close 35718 (time 1746137213.97 total 269.50 last 133.27) - open/close 37235 (time 1746137223.97 total 279.51 last 151.61) - open/close 38618 (time 1746137233.97 total 289.51 last 138.28) - open/close 40000 (time 1746137243.70 total 299.24 last 142.12) - open/close 41349 (time 1746137253.71 total 309.24 last 134.80) - open/close 42934 (time 1746137263.71 total 319.24 last 158.47) - open/close 44596 (time 1746137273.71 total 329.25 last 166.13) - open/close 46012 (time 1746137283.72 total 339.26 last 141.48) - open/close 47464 (time 1746137293.72 total 349.26 last 145.15) - open/close 49113 (time 1746137303.72 total 359.26 last 164.90) - open/close 50000 (time 1746137309.86 total 365.39 last 144.59) - open/close 51381 (time 1746137319.86 total 375.40 last 138.07) - open/close 52960 (time 1746137329.86 total 385.40 last 157.87) - open/close 54543 (time 1746137339.86 total 395.40 last 158.27) - open/close 56055 (time 1746137349.87 total 405.40 last 151.17) - open/close 57571 (time 1746137359.87 total 415.41 last 151.54) - open/close 59246 (time 1746137369.87 total 425.41 last 167.47) - open/close 60000 (time 1746137374.97 total 430.51 last 147.77) - open/close 61759 (time 1746137384.98 total 440.51 last 175.83) - open/close 63385 (time 1746137394.98 total 450.52 last 162.56) total: 64768 open/close in 459.46 seconds: 140.97 ops/second - /unlink 2772 (time 1746137415.48 total 10.00 last 277.16) - /unlink 5796 (time 1746137425.49 total 20.01 last 302.23) - /unlink 8209 (time 1746137435.50 total 30.01 last 241.17) - /unlink 10000 (time 1746137443.15 total 37.66 last 234.04) - /unlink 12591 (time 1746137453.15 total 47.67 last 259.03) - /unlink 15073 (time 1746137463.15 total 57.67 last 248.20) - /unlink 17593 (time 1746137473.15 total 67.67 last 251.94) - /unlink 20000 (time 1746137482.88 total 77.40 last 247.43) - /unlink 22766 (time 1746137492.89 total 87.40 last 276.43) - /unlink 25423 (time 1746137502.89 total 97.41 last 265.65) - /unlink 28702 (time 1746137512.89 total 107.41 last 327.84) - /unlink 30000 (time 1746137516.86 total 111.38 last 326.72) - /unlink 33636 (time 1746137526.87 total 121.38 last 363.55) - /unlink 37108 (time 1746137536.87 total 131.39 last 347.04) - /unlink 39728 (time 1746137546.87 total 141.39 last 261.97) - /unlink 42196 (time 1746137556.87 total 151.39 last 246.72) - /unlink 44357 (time 1746137566.88 total 161.40 last 216.02) - /unlink 46941 (time 1746137576.88 total 171.40 last 258.39) - /unlink 49482 (time 1746137586.88 total 181.40 last 254.01) - /unlink 52157 (time 1746137596.88 total 191.40 last 267.45) - /unlink 54564 (time 1746137606.89 total 201.40 last 240.64) - /unlink 57418 (time 1746137616.89 total 211.41 last 285.32) - /unlink 59923 (time 1746137626.89 total 221.41 last 250.44) - /unlink 63042 (time 1746137636.89 total 231.41 last 311.82) total: 64768 /unlink in 238.83 seconds: 271.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1167 (time 1746137671.17 total 10.00 last 116.68) - open/close 2599 (time 1746137681.17 total 20.01 last 143.13) - open/close 4196 (time 1746137691.18 total 30.01 last 159.57) - open/close 5678 (time 1746137701.18 total 40.02 last 148.14) - open/close 7339 (time 1746137711.19 total 50.02 last 166.05) - open/close 8850 (time 1746137721.19 total 60.02 last 151.08) - open/close 10000 (time 1746137728.46 total 67.30 last 158.11) - open/close 11695 (time 1746137738.47 total 77.30 last 169.39) - open/close 13135 (time 1746137748.47 total 87.30 last 143.97) - open/close 14896 (time 1746137758.47 total 97.31 last 176.06) - open/close 16539 (time 1746137768.47 total 107.31 last 164.26) - open/close 18108 (time 1746137778.48 total 117.32 last 156.77) - open/close 19751 (time 1746137788.48 total 127.32 last 164.26) - open/close 21666 (time 1746137798.49 total 137.33 last 191.33) - open/close 23167 (time 1746137808.50 total 147.34 last 149.98) - open/close 24489 (time 1746137818.50 total 157.34 last 132.15) - open/close 25937 (time 1746137828.51 total 167.35 last 144.69) - open/close 27165 (time 1746137838.52 total 177.35 last 122.76) - open/close 28915 (time 1746137848.52 total 187.36 last 174.90) - open/close 30000 (time 1746137855.53 total 194.37 last 154.73) - open/close 31620 (time 1746137865.54 total 204.38 last 161.85) - open/close 33001 (time 1746137875.54 total 214.38 last 138.07) - open/close 34349 (time 1746137885.55 total 224.39 last 134.72) - open/close 36152 (time 1746137895.55 total 234.39 last 180.26) - open/close 37547 (time 1746137905.56 total 244.40 last 139.39) - open/close 39179 (time 1746137915.56 total 254.40 last 163.16) - open/close 40000 (time 1746137920.44 total 259.27 last 168.35) - open/close 41849 (time 1746137930.44 total 269.28 last 184.83) - open/close 43841 (time 1746137940.45 total 279.28 last 199.16) - open/close 45646 (time 1746137950.45 total 289.28 last 180.44) - open/close 47502 (time 1746137960.45 total 299.29 last 185.58) - open/close 49640 (time 1746137970.45 total 309.29 last 213.73) - open/close 51730 (time 1746137980.45 total 319.29 last 208.99) - open/close 53717 (time 1746137990.46 total 329.30 last 198.57) - open/close 55359 (time 1746138000.47 total 339.30 last 164.12) - open/close 57171 (time 1746138010.47 total 349.30 last 181.13) - open/close 58933 (time 1746138020.47 total 359.30 last 176.20) - open/close 60000 (time 1746138025.91 total 364.74 last 196.22) - open/close 61992 (time 1746138035.91 total 374.74 last 199.19) - open/close 63574 (time 1746138045.92 total 384.75 last 158.06) total: 64768 open/close in 392.40 seconds: 165.06 ops/second - /unlink 2801 (time 1746138064.78 total 10.00 last 280.01) - /unlink 5044 (time 1746138074.79 total 20.01 last 224.14) - /unlink 8289 (time 1746138084.79 total 30.01 last 324.44) - /unlink 10000 (time 1746138090.17 total 35.39 last 317.90) - /unlink 12948 (time 1746138100.18 total 45.40 last 294.66) - /unlink 16240 (time 1746138110.18 total 55.40 last 329.17) - /unlink 19134 (time 1746138120.18 total 65.40 last 289.31) - /unlink 20000 (time 1746138122.44 total 67.66 last 384.13) - /unlink 23570 (time 1746138132.44 total 77.66 last 356.90) - /unlink 27432 (time 1746138142.44 total 87.66 last 386.14) - /unlink 30000 (time 1746138151.72 total 96.94 last 276.69) - /unlink 32800 (time 1746138161.72 total 106.95 last 279.95) - /unlink 35504 (time 1746138171.73 total 116.95 last 270.36) - /unlink 38378 (time 1746138181.73 total 126.95 last 287.36) - /unlink 40000 (time 1746138188.14 total 133.36 last 252.87) - /unlink 42572 (time 1746138198.14 total 143.36 last 257.18) - /unlink 45177 (time 1746138208.14 total 153.37 last 260.45) - /unlink 48303 (time 1746138218.15 total 163.37 last 312.54) - /unlink 50000 (time 1746138223.65 total 168.88 last 308.09) - /unlink 53547 (time 1746138233.66 total 178.88 last 354.66) - /unlink 57841 (time 1746138243.66 total 188.88 last 429.29) - /unlink 60000 (time 1746138249.11 total 194.33 last 395.94) - /unlink 63064 (time 1746138259.11 total 204.33 last 306.39) total: 64768 /unlink in 208.63 seconds: 310.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1826 (time 1746138290.98 total 10.01 last 182.50) - open/close 3257 (time 1746138300.99 total 20.01 last 143.04) - open/close 4661 (time 1746138310.99 total 30.01 last 140.35) - open/close 6103 (time 1746138321.00 total 40.02 last 144.11) - open/close 7523 (time 1746138331.00 total 50.02 last 141.96) - open/close 8761 (time 1746138341.00 total 60.03 last 123.75) - open/close 10000 (time 1746138348.85 total 67.88 last 157.81) - open/close 11728 (time 1746138358.86 total 77.88 last 172.71) - open/close 13437 (time 1746138368.86 total 87.88 last 170.88) - open/close 14901 (time 1746138378.86 total 97.89 last 146.35) - open/close 16617 (time 1746138388.87 total 107.89 last 171.54) - open/close 18534 (time 1746138398.87 total 117.89 last 191.66) - open/close 20000 (time 1746138406.29 total 125.31 last 197.59) - open/close 22216 (time 1746138416.29 total 135.32 last 221.51) - open/close 23945 (time 1746138426.30 total 145.32 last 172.85) - open/close 25931 (time 1746138436.30 total 155.32 last 198.59) - open/close 27921 (time 1746138446.30 total 165.32 last 198.98) - open/close 29725 (time 1746138456.31 total 175.33 last 180.27) - open/close 31446 (time 1746138466.31 total 185.33 last 172.08) - open/close 33376 (time 1746138476.31 total 195.33 last 192.93) - open/close 34972 (time 1746138486.31 total 205.34 last 159.56) - open/close 36595 (time 1746138496.32 total 215.34 last 162.23) - open/close 38037 (time 1746138506.32 total 225.34 last 144.15) - open/close 39504 (time 1746138516.32 total 235.35 last 146.66) - open/close 40000 (time 1746138519.72 total 238.74 last 146.05) - open/close 41478 (time 1746138529.72 total 248.74 last 147.77) - open/close 42879 (time 1746138539.73 total 258.76 last 139.92) - open/close 44479 (time 1746138549.74 total 268.76 last 159.88) - open/close 45757 (time 1746138559.74 total 278.77 last 127.78) - open/close 47355 (time 1746138569.75 total 288.77 last 159.75) - open/close 48867 (time 1746138579.75 total 298.77 last 151.15) - open/close 50000 (time 1746138588.25 total 307.27 last 133.36) - open/close 51322 (time 1746138598.25 total 317.27 last 132.17) - open/close 52876 (time 1746138608.25 total 327.27 last 155.35) - open/close 54068 (time 1746138618.26 total 337.28 last 119.15) - open/close 55538 (time 1746138628.26 total 347.28 last 146.98) - open/close 57115 (time 1746138638.26 total 357.29 last 157.59) - open/close 58870 (time 1746138648.27 total 367.29 last 175.49) - open/close 60000 (time 1746138653.61 total 372.63 last 211.51) - open/close 61649 (time 1746138663.61 total 382.64 last 164.79) - open/close 63275 (time 1746138673.62 total 392.64 last 162.54) total: 64768 open/close in 402.25 seconds: 161.01 ops/second - /unlink 2134 (time 1746138694.71 total 10.00 last 213.31) - /unlink 4449 (time 1746138704.71 total 20.01 last 231.47) - /unlink 6396 (time 1746138714.71 total 30.01 last 194.65) - /unlink 8233 (time 1746138724.71 total 40.01 last 183.68) - /unlink 10000 (time 1746138733.35 total 48.65 last 204.58) - /unlink 12449 (time 1746138743.35 total 58.65 last 244.82) - /unlink 14921 (time 1746138753.36 total 68.65 last 247.14) - /unlink 17425 (time 1746138763.36 total 78.65 last 250.36) - /unlink 19485 (time 1746138773.36 total 88.65 last 205.98) - /unlink 20000 (time 1746138775.85 total 91.15 last 206.76) - /unlink 22706 (time 1746138785.85 total 101.15 last 270.50) - /unlink 25075 (time 1746138795.86 total 111.15 last 236.82) - /unlink 27206 (time 1746138805.86 total 121.15 last 213.09) - /unlink 29498 (time 1746138815.86 total 131.15 last 229.15) - /unlink 30000 (time 1746138818.40 total 133.69 last 197.67) - /unlink 32125 (time 1746138828.40 total 143.70 last 212.44) - /unlink 34314 (time 1746138838.40 total 153.70 last 218.90) - /unlink 36685 (time 1746138848.40 total 163.70 last 237.08) - /unlink 38856 (time 1746138858.41 total 173.70 last 217.02) - /unlink 40000 (time 1746138863.32 total 178.62 last 232.58) - /unlink 42099 (time 1746138873.33 total 188.62 last 209.85) - /unlink 44075 (time 1746138883.33 total 198.62 last 197.58) - /unlink 46211 (time 1746138893.33 total 208.62 last 213.58) - /unlink 48090 (time 1746138903.33 total 218.63 last 187.84) - /unlink 50000 (time 1746138912.83 total 228.13 last 201.08) - /unlink 51779 (time 1746138922.83 total 238.13 last 177.86) - /unlink 53557 (time 1746138932.83 total 248.13 last 177.77) - /unlink 55955 (time 1746138942.84 total 258.13 last 239.76) - /unlink 58288 (time 1746138952.84 total 268.14 last 233.19) - /unlink 60000 (time 1746138959.77 total 275.07 last 247.01) - /unlink 62428 (time 1746138969.77 total 285.07 last 242.79) - /unlink 64286 (time 1746138979.78 total 295.07 last 185.66) total: 64768 /unlink in 297.76 seconds: 217.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1293 (time 1746139010.75 total 10.01 last 129.20) - open/close 2534 (time 1746139020.76 total 20.02 last 123.96) - open/close 3957 (time 1746139030.76 total 30.02 last 142.30) - open/close 5179 (time 1746139040.77 total 40.03 last 122.12) - open/close 6476 (time 1746139050.77 total 50.03 last 129.62) - open/close 7702 (time 1746139060.78 total 60.03 last 122.56) - open/close 8715 (time 1746139070.79 total 70.04 last 101.21) - open/close 9877 (time 1746139080.79 total 80.05 last 116.17) - open/close 11195 (time 1746139090.79 total 90.05 last 131.76) - open/close 12646 (time 1746139100.80 total 100.05 last 145.01) - open/close 14068 (time 1746139110.80 total 110.06 last 142.17) - open/close 15821 (time 1746139120.80 total 120.06 last 175.22) - open/close 17764 (time 1746139130.81 total 130.07 last 194.18) - open/close 19049 (time 1746139140.81 total 140.07 last 128.45) - open/close 20000 (time 1746139148.43 total 147.69 last 124.84) - open/close 21349 (time 1746139158.44 total 157.70 last 134.80) - open/close 22911 (time 1746139168.44 total 167.70 last 156.16) - open/close 24243 (time 1746139178.45 total 177.71 last 133.03) - open/close 25326 (time 1746139188.46 total 187.72 last 108.26) - open/close 26678 (time 1746139198.46 total 197.72 last 135.14) - open/close 27993 (time 1746139208.47 total 207.73 last 131.43) - open/close 29183 (time 1746139218.47 total 217.73 last 118.99) - open/close 30000 (time 1746139224.61 total 223.87 last 133.10) - open/close 31336 (time 1746139234.61 total 233.87 last 133.58) - open/close 32572 (time 1746139244.62 total 243.87 last 123.51) - open/close 33948 (time 1746139254.62 total 253.88 last 137.57) - open/close 35154 (time 1746139264.62 total 263.88 last 120.57) - open/close 36696 (time 1746139274.62 total 273.88 last 154.15) - open/close 38362 (time 1746139284.63 total 283.89 last 166.44) - open/close 39610 (time 1746139294.64 total 293.90 last 124.75) - open/close 40000 (time 1746139298.43 total 297.69 last 102.78) - open/close 41063 (time 1746139308.44 total 307.70 last 106.22) - open/close 42160 (time 1746139318.45 total 317.71 last 109.60) - open/close 43536 (time 1746139328.45 total 327.71 last 137.58) - open/close 44985 (time 1746139338.46 total 337.72 last 144.78) - open/close 46245 (time 1746139348.46 total 347.72 last 125.95) - open/close 47473 (time 1746139358.46 total 357.72 last 122.77) - open/close 48563 (time 1746139368.48 total 367.73 last 108.88) - open/close 49671 (time 1746139378.48 total 377.74 last 110.75) - open/close 50000 (time 1746139381.11 total 380.37 last 125.14) - open/close 51142 (time 1746139391.12 total 390.37 last 114.14) - open/close 52559 (time 1746139401.12 total 400.38 last 141.65) - open/close 53841 (time 1746139411.12 total 410.38 last 128.18) - open/close 55314 (time 1746139421.12 total 420.38 last 147.25) - open/close 56525 (time 1746139431.13 total 430.39 last 121.01) - open/close 57697 (time 1746139441.14 total 440.40 last 117.10) - open/close 58760 (time 1746139451.15 total 450.40 last 106.24) - open/close 59889 (time 1746139461.15 total 460.41 last 112.82) - open/close 60945 (time 1746139471.16 total 470.42 last 105.55) - open/close 62093 (time 1746139481.16 total 480.42 last 114.73) - open/close 63055 (time 1746139491.17 total 490.42 last 96.19) - open/close 64082 (time 1746139501.17 total 500.43 last 102.66) total: 64768 open/close in 506.74 seconds: 127.81 ops/second - /unlink 2102 (time 1746139519.34 total 10.00 last 210.19) - /unlink 4196 (time 1746139529.34 total 20.00 last 209.37) - /unlink 6006 (time 1746139539.35 total 30.01 last 180.94) - /unlink 8091 (time 1746139549.35 total 40.01 last 208.49) - /unlink 10000 (time 1746139555.76 total 46.42 last 297.64) - /unlink 12728 (time 1746139565.76 total 56.42 last 272.74) - /unlink 14988 (time 1746139575.76 total 66.42 last 225.98) - /unlink 17993 (time 1746139585.77 total 76.42 last 300.43) - /unlink 20000 (time 1746139592.48 total 83.14 last 298.89) - /unlink 21734 (time 1746139602.48 total 93.14 last 173.34) - /unlink 23436 (time 1746139612.49 total 103.14 last 170.17) - /unlink 25867 (time 1746139622.49 total 113.15 last 242.99) - /unlink 28102 (time 1746139632.49 total 123.15 last 223.48) - /unlink 29827 (time 1746139642.49 total 133.15 last 172.49) - /unlink 31842 (time 1746139652.50 total 143.16 last 201.37) - /unlink 34097 (time 1746139662.50 total 153.16 last 225.46) - /unlink 36084 (time 1746139672.51 total 163.17 last 198.51) - /unlink 38194 (time 1746139682.52 total 173.18 last 210.82) - /unlink 40000 (time 1746139691.17 total 181.83 last 208.73) - /unlink 41915 (time 1746139701.17 total 191.83 last 191.46) - /unlink 43809 (time 1746139711.18 total 201.83 last 189.35) - /unlink 45802 (time 1746139721.18 total 211.84 last 199.26) - /unlink 47667 (time 1746139731.18 total 221.84 last 186.49) - /unlink 49402 (time 1746139741.18 total 231.84 last 173.43) - /unlink 50000 (time 1746139744.41 total 235.06 last 185.55) - /unlink 52194 (time 1746139754.41 total 245.07 last 219.37) - /unlink 54341 (time 1746139764.41 total 255.07 last 214.66) - /unlink 56518 (time 1746139774.41 total 265.07 last 217.64) - /unlink 58551 (time 1746139784.41 total 275.07 last 203.26) - /unlink 60000 (time 1746139789.84 total 280.50 last 266.86) - /unlink 62197 (time 1746139799.84 total 290.50 last 219.68) - /unlink 64107 (time 1746139809.85 total 300.51 last 190.95) total: 64768 /unlink in 303.37 seconds: 213.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1293 (time 1746139842.21 total 10.00 last 129.30) - open/close 2713 (time 1746139852.21 total 20.00 last 141.99) - open/close 4338 (time 1746139862.22 total 30.00 last 162.45) - open/close 5859 (time 1746139872.22 total 40.01 last 152.00) - open/close 7305 (time 1746139882.22 total 50.01 last 144.60) - open/close 8926 (time 1746139892.23 total 60.02 last 162.00) - open/close 10000 (time 1746139899.42 total 67.21 last 149.33) - open/close 11526 (time 1746139909.42 total 77.21 last 152.55) - open/close 12796 (time 1746139919.43 total 87.21 last 126.98) - open/close 14047 (time 1746139929.43 total 97.22 last 125.08) - open/close 15436 (time 1746139939.43 total 107.22 last 138.88) - open/close 16665 (time 1746139949.43 total 117.22 last 122.89) - open/close 17836 (time 1746139959.43 total 127.22 last 117.07) - open/close 18958 (time 1746139969.44 total 137.23 last 112.14) - open/close 20000 (time 1746139978.96 total 146.75 last 109.39) - open/close 21176 (time 1746139988.97 total 156.76 last 117.51) - open/close 22454 (time 1746139998.98 total 166.77 last 127.66) - open/close 23673 (time 1746140009.00 total 176.79 last 121.63) - open/close 24872 (time 1746140019.01 total 186.80 last 119.79) - open/close 25912 (time 1746140029.02 total 196.81 last 103.88) - open/close 27029 (time 1746140039.02 total 206.81 last 111.70) - open/close 28138 (time 1746140049.03 total 216.82 last 110.87) - open/close 29345 (time 1746140059.03 total 226.82 last 120.67) - open/close 30000 (time 1746140065.19 total 232.98 last 106.28) - open/close 30996 (time 1746140075.19 total 242.98 last 99.58) - open/close 32100 (time 1746140085.21 total 252.99 last 110.29) - open/close 33282 (time 1746140095.21 total 263.00 last 118.15) - open/close 34477 (time 1746140105.21 total 273.00 last 119.50) - open/close 35650 (time 1746140115.22 total 283.01 last 117.20) - open/close 36760 (time 1746140125.22 total 293.01 last 111.00) - open/close 37797 (time 1746140135.23 total 303.02 last 103.58) - open/close 38728 (time 1746140145.23 total 313.02 last 93.09) - open/close 39801 (time 1746140155.23 total 323.02 last 107.27) - open/close 40000 (time 1746140157.28 total 325.07 last 97.29) - open/close 41055 (time 1746140167.28 total 335.07 last 105.50) - open/close 42108 (time 1746140177.28 total 345.07 last 105.24) - open/close 43197 (time 1746140187.29 total 355.08 last 108.85) - open/close 44408 (time 1746140197.29 total 365.08 last 121.07) - open/close 45428 (time 1746140207.30 total 375.09 last 101.90) - open/close 46570 (time 1746140217.30 total 385.09 last 114.17) - open/close 47699 (time 1746140227.31 total 395.10 last 112.81) - open/close 48817 (time 1746140237.32 total 405.11 last 111.74) - open/close 49884 (time 1746140247.32 total 415.11 last 106.64) - open/close 50960 (time 1746140257.33 total 425.12 last 107.55) - open/close 51967 (time 1746140267.33 total 435.12 last 100.63) - open/close 53048 (time 1746140277.33 total 445.12 last 108.09) - open/close 54162 (time 1746140287.34 total 455.13 last 111.36) - open/close 55273 (time 1746140297.34 total 465.13 last 111.04) - open/close 56307 (time 1746140307.35 total 475.14 last 103.38) - open/close 57394 (time 1746140317.35 total 485.14 last 108.66) - open/close 58408 (time 1746140327.35 total 495.14 last 101.38) - open/close 59440 (time 1746140337.36 total 505.14 last 103.17) - open/close 60000 (time 1746140342.47 total 510.25 last 109.59) - open/close 61100 (time 1746140352.47 total 520.26 last 109.96) - open/close 62194 (time 1746140362.47 total 530.26 last 109.40) - open/close 63297 (time 1746140372.48 total 540.27 last 110.19) - open/close 64413 (time 1746140382.48 total 550.27 last 111.58) total: 64768 open/close in 553.50 seconds: 117.02 ops/second - /unlink 1879 (time 1746140397.79 total 10.00 last 187.85) - /unlink 3635 (time 1746140407.80 total 20.01 last 175.47) - /unlink 5089 (time 1746140417.80 total 30.01 last 145.36) - /unlink 6769 (time 1746140427.81 total 40.02 last 167.90) - /unlink 8380 (time 1746140437.81 total 50.02 last 161.05) - /unlink 10000 (time 1746140447.09 total 59.30 last 174.64) - /unlink 11788 (time 1746140457.09 total 69.30 last 178.78) - /unlink 13742 (time 1746140467.09 total 79.30 last 195.37) - /unlink 15599 (time 1746140477.09 total 89.31 last 185.61) - /unlink 17460 (time 1746140487.10 total 99.31 last 186.08) - /unlink 19245 (time 1746140497.10 total 109.31 last 178.42) - /unlink 20000 (time 1746140501.31 total 113.52 last 179.52) - /unlink 21778 (time 1746140511.31 total 123.52 last 177.78) - /unlink 23454 (time 1746140521.31 total 133.52 last 167.53) - /unlink 25197 (time 1746140531.31 total 143.52 last 174.28) - /unlink 26947 (time 1746140541.31 total 153.52 last 174.99) - /unlink 28877 (time 1746140551.32 total 163.53 last 192.94) - /unlink 30000 (time 1746140557.64 total 169.85 last 177.49) - /unlink 31844 (time 1746140567.64 total 179.85 last 184.40) - /unlink 33614 (time 1746140577.65 total 189.86 last 176.89) - /unlink 35444 (time 1746140587.66 total 199.87 last 182.85) - /unlink 37192 (time 1746140597.66 total 209.87 last 174.73) - /unlink 38864 (time 1746140607.66 total 219.87 last 167.19) - /unlink 40000 (time 1746140614.04 total 226.25 last 178.14) - /unlink 41841 (time 1746140624.04 total 236.25 last 184.05) - /unlink 43489 (time 1746140634.05 total 246.26 last 164.68) - /unlink 45196 (time 1746140644.05 total 256.27 last 170.60) - /unlink 47401 (time 1746140654.06 total 266.27 last 220.48) - /unlink 49523 (time 1746140664.06 total 276.27 last 212.19) - /unlink 51664 (time 1746140674.06 total 286.27 last 214.10) - /unlink 54587 (time 1746140684.06 total 296.27 last 292.25) - /unlink 56967 (time 1746140694.06 total 306.27 last 238.00) - /unlink 59310 (time 1746140704.06 total 316.27 last 234.19) - /unlink 60000 (time 1746140707.73 total 319.94 last 188.34) - /unlink 61736 (time 1746140717.73 total 329.94 last 173.50) - /unlink 63882 (time 1746140727.74 total 339.95 last 214.51) total: 64768 /unlink in 344.64 seconds: 187.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2