-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 22:18:25 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg250-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg250-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg250-server oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg250-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg250-server: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg250-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:19:46 (1746065986) Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg250-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1313 (time 1746066040.49 total 10.00 last 131.25) - open/close 2614 (time 1746066050.50 total 20.01 last 129.99) - open/close 3918 (time 1746066060.50 total 30.01 last 130.37) - open/close 5383 (time 1746066070.50 total 40.02 last 146.48) - open/close 6911 (time 1746066080.50 total 50.02 last 152.73) - open/close 8376 (time 1746066090.51 total 60.03 last 146.41) - open/close 9792 (time 1746066100.52 total 70.03 last 141.54) - open/close 11181 (time 1746066110.52 total 80.03 last 138.86) - open/close 12600 (time 1746066120.52 total 90.04 last 141.85) - open/close 13952 (time 1746066130.53 total 100.05 last 135.09) - open/close 15190 (time 1746066140.53 total 110.05 last 123.75) - open/close 16505 (time 1746066150.54 total 120.06 last 131.42) - open/close 17666 (time 1746066160.55 total 130.06 last 116.03) - open/close 19024 (time 1746066170.55 total 140.07 last 135.73) - open/close 20000 (time 1746066178.07 total 147.59 last 129.82) - open/close 21270 (time 1746066188.07 total 157.59 last 126.99) - open/close 22574 (time 1746066198.07 total 167.59 last 130.35) - open/close 23934 (time 1746066208.08 total 177.59 last 135.97) - open/close 25328 (time 1746066218.08 total 187.59 last 139.39) - open/close 26641 (time 1746066228.09 total 197.61 last 131.07) - open/close 27915 (time 1746066238.10 total 207.62 last 127.30) - open/close 29274 (time 1746066248.10 total 217.62 last 135.89) - open/close 30000 (time 1746066252.95 total 222.46 last 149.90) - open/close 31405 (time 1746066262.96 total 232.47 last 140.35) - open/close 32782 (time 1746066272.96 total 242.48 last 137.66) - open/close 34150 (time 1746066282.96 total 252.48 last 136.79) - open/close 35417 (time 1746066292.96 total 262.48 last 126.66) - open/close 36810 (time 1746066302.97 total 272.48 last 139.24) - open/close 38447 (time 1746066312.97 total 282.49 last 163.64) - open/close 39973 (time 1746066322.97 total 292.49 last 152.55) - open/close 41709 (time 1746066332.98 total 302.49 last 173.53) - open/close 43550 (time 1746066342.98 total 312.50 last 184.04) - open/close 45093 (time 1746066352.99 total 322.50 last 154.20) - open/close 46544 (time 1746066362.99 total 332.51 last 145.05) - open/close 48143 (time 1746066372.99 total 342.51 last 159.88) - open/close 49486 (time 1746066383.00 total 352.52 last 134.20) - open/close 50000 (time 1746066387.21 total 356.73 last 121.97) - open/close 51256 (time 1746066397.22 total 366.74 last 125.49) - open/close 52389 (time 1746066407.22 total 376.74 last 113.30) - open/close 53593 (time 1746066417.23 total 386.75 last 120.28) - open/close 54845 (time 1746066427.23 total 396.75 last 125.18) - open/close 56183 (time 1746066437.24 total 406.75 last 133.79) - open/close 57562 (time 1746066447.24 total 416.75 last 137.87) - open/close 59014 (time 1746066457.24 total 426.76 last 145.16) - open/close 60000 (time 1746066464.14 total 433.66 last 142.92) - open/close 61208 (time 1746066474.14 total 443.66 last 120.75) - open/close 62256 (time 1746066484.15 total 453.67 last 104.73) - open/close 63346 (time 1746066494.15 total 463.67 last 108.98) - open/close 64709 (time 1746066504.16 total 473.67 last 136.23) total: 64768 open/close in 474.13 seconds: 136.60 ops/second - /unlink 2015 (time 1746066516.37 total 10.00 last 201.42) - /unlink 4211 (time 1746066526.38 total 20.01 last 219.56) - /unlink 6467 (time 1746066536.38 total 30.01 last 225.51) - /unlink 8482 (time 1746066546.38 total 40.01 last 201.43) - /unlink 10000 (time 1746066555.16 total 48.79 last 172.93) - /unlink 11515 (time 1746066565.16 total 58.80 last 151.44) - /unlink 12975 (time 1746066575.17 total 68.80 last 145.99) - /unlink 14522 (time 1746066585.17 total 78.80 last 154.66) - /unlink 15972 (time 1746066595.17 total 88.80 last 144.99) - /unlink 17644 (time 1746066605.17 total 98.80 last 167.16) - /unlink 19461 (time 1746066615.17 total 108.80 last 181.69) - /unlink 20000 (time 1746066617.82 total 111.45 last 203.67) - /unlink 21711 (time 1746066627.82 total 121.45 last 171.05) - /unlink 23442 (time 1746066637.82 total 131.45 last 173.07) - /unlink 25283 (time 1746066647.83 total 141.46 last 184.05) - /unlink 27153 (time 1746066657.83 total 151.46 last 186.87) - /unlink 28842 (time 1746066667.84 total 161.47 last 168.76) - /unlink 30000 (time 1746066675.14 total 168.77 last 158.61) - /unlink 31872 (time 1746066685.14 total 178.77 last 187.15) - /unlink 34071 (time 1746066695.15 total 188.78 last 219.82) - /unlink 36155 (time 1746066705.15 total 198.78 last 208.34) - /unlink 38371 (time 1746066715.15 total 208.78 last 221.60) - /unlink 40000 (time 1746066721.58 total 215.21 last 253.51) - /unlink 42331 (time 1746066731.58 total 225.21 last 233.08) - /unlink 44379 (time 1746066741.58 total 235.21 last 204.75) - /unlink 46284 (time 1746066751.58 total 245.21 last 190.44) - /unlink 48253 (time 1746066761.59 total 255.22 last 196.85) - /unlink 50000 (time 1746066770.42 total 264.05 last 197.72) - /unlink 51898 (time 1746066780.42 total 274.05 last 189.76) - /unlink 53885 (time 1746066790.43 total 284.06 last 198.63) - /unlink 55811 (time 1746066800.43 total 294.06 last 192.54) - /unlink 57640 (time 1746066810.43 total 304.06 last 182.88) - /unlink 60000 (time 1746066819.69 total 313.33 last 254.76) - /unlink 62067 (time 1746066829.69 total 323.33 last 206.70) - /unlink 64169 (time 1746066839.70 total 333.33 last 210.19) total: 64768 /unlink in 336.21 seconds: 192.64 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1196 (time 1746066873.10 total 10.00 last 119.59) - open/close 2545 (time 1746066883.11 total 20.01 last 134.84) - open/close 3716 (time 1746066893.11 total 30.01 last 117.08) - open/close 4961 (time 1746066903.11 total 40.01 last 124.45) - open/close 6258 (time 1746066913.12 total 50.01 last 129.66) - open/close 7959 (time 1746066923.12 total 60.02 last 170.05) - open/close 9377 (time 1746066933.12 total 70.02 last 141.74) - open/close 10000 (time 1746066938.35 total 75.24 last 119.30) - open/close 11326 (time 1746066948.35 total 85.25 last 132.54) - open/close 12663 (time 1746066958.35 total 95.25 last 133.66) - open/close 13891 (time 1746066968.36 total 105.26 last 122.73) - open/close 15053 (time 1746066978.37 total 115.26 last 116.12) - open/close 16175 (time 1746066988.37 total 125.26 last 112.19) - open/close 17304 (time 1746066998.37 total 135.27 last 112.84) - open/close 18593 (time 1746067008.38 total 145.27 last 128.83) - open/close 20000 (time 1746067017.87 total 154.77 last 148.20) - open/close 21278 (time 1746067027.87 total 164.77 last 127.76) - open/close 22639 (time 1746067037.89 total 174.79 last 135.92) - open/close 23928 (time 1746067047.89 total 184.79 last 128.80) - open/close 25285 (time 1746067057.90 total 194.80 last 135.67) - open/close 26684 (time 1746067067.90 total 204.80 last 139.80) - open/close 27967 (time 1746067077.90 total 214.80 last 128.30) - open/close 29570 (time 1746067087.91 total 224.81 last 160.26) - open/close 30000 (time 1746067090.52 total 227.42 last 164.73) - open/close 31980 (time 1746067100.52 total 237.42 last 197.88) - open/close 33736 (time 1746067110.52 total 247.42 last 175.59) - open/close 35319 (time 1746067120.53 total 257.43 last 158.24) - open/close 36932 (time 1746067130.53 total 267.43 last 161.29) - open/close 38726 (time 1746067140.53 total 277.43 last 179.35) - open/close 40000 (time 1746067146.68 total 283.58 last 207.31) - open/close 41440 (time 1746067156.68 total 293.58 last 143.99) - open/close 42635 (time 1746067166.69 total 303.58 last 119.40) - open/close 43823 (time 1746067176.69 total 313.59 last 118.75) - open/close 45053 (time 1746067186.70 total 323.59 last 122.93) - open/close 46439 (time 1746067196.70 total 333.60 last 138.54) - open/close 47869 (time 1746067206.70 total 343.60 last 142.97) - open/close 49253 (time 1746067216.70 total 353.60 last 138.37) - open/close 50000 (time 1746067222.37 total 359.27 last 131.89) - open/close 51339 (time 1746067232.38 total 369.27 last 133.82) - open/close 52651 (time 1746067242.38 total 379.28 last 131.14) - open/close 54025 (time 1746067252.38 total 389.28 last 137.36) - open/close 55613 (time 1746067262.39 total 399.29 last 158.70) - open/close 56867 (time 1746067272.40 total 409.29 last 125.30) - open/close 58102 (time 1746067282.40 total 419.30 last 123.45) - open/close 59485 (time 1746067292.40 total 429.30 last 138.28) - open/close 60000 (time 1746067295.59 total 432.49 last 161.41) - open/close 61445 (time 1746067305.59 total 442.49 last 144.49) - open/close 62893 (time 1746067315.59 total 452.49 last 144.78) - open/close 64278 (time 1746067325.59 total 462.49 last 138.49) total: 64768 open/close in 466.23 seconds: 138.92 ops/second - /unlink 2940 (time 1746067340.77 total 10.00 last 294.00) - /unlink 6051 (time 1746067350.77 total 20.00 last 311.03) - /unlink 9269 (time 1746067360.78 total 30.00 last 321.74) - /unlink 10000 (time 1746067363.14 total 32.37 last 309.58) - /unlink 12514 (time 1746067373.14 total 42.37 last 251.38) - /unlink 14552 (time 1746067383.14 total 52.37 last 203.76) - /unlink 16858 (time 1746067393.14 total 62.37 last 230.58) - /unlink 19321 (time 1746067403.14 total 72.37 last 246.28) - /unlink 20000 (time 1746067406.24 total 75.47 last 219.19) - /unlink 22336 (time 1746067416.24 total 85.47 last 233.52) - /unlink 24553 (time 1746067426.25 total 95.48 last 221.49) - /unlink 27146 (time 1746067436.25 total 105.48 last 259.24) - /unlink 29731 (time 1746067446.26 total 115.48 last 258.48) - /unlink 32407 (time 1746067456.26 total 125.49 last 267.52) - /unlink 35294 (time 1746067466.26 total 135.49 last 288.66) - /unlink 37749 (time 1746067476.26 total 145.49 last 245.47) - /unlink 40000 (time 1746067485.02 total 154.25 last 257.01) - /unlink 42502 (time 1746067495.02 total 164.25 last 250.16) - /unlink 44836 (time 1746067505.02 total 174.25 last 233.40) - /unlink 47464 (time 1746067515.02 total 184.25 last 262.78) - /unlink 50000 (time 1746067523.22 total 192.45 last 309.25) - /unlink 52697 (time 1746067533.22 total 202.45 last 269.70) - /unlink 55458 (time 1746067543.22 total 212.45 last 276.04) - /unlink 57849 (time 1746067553.23 total 222.45 last 239.08) - /unlink 60000 (time 1746067560.51 total 229.74 last 295.20) - /unlink 62880 (time 1746067570.51 total 239.74 last 287.96) total: 64768 /unlink in 245.35 seconds: 263.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1610 (time 1746067604.13 total 10.01 last 160.83) - open/close 3273 (time 1746067614.14 total 20.02 last 166.15) - open/close 4943 (time 1746067624.14 total 30.03 last 166.90) - open/close 6656 (time 1746067634.14 total 40.03 last 171.29) - open/close 8319 (time 1746067644.15 total 50.03 last 166.23) - open/close 9776 (time 1746067654.16 total 60.04 last 145.53) - open/close 11384 (time 1746067664.16 total 70.04 last 160.76) - open/close 12813 (time 1746067674.16 total 80.05 last 142.89) - open/close 14784 (time 1746067684.16 total 90.05 last 197.09) - open/close 16784 (time 1746067694.17 total 100.06 last 199.81) - open/close 18482 (time 1746067704.18 total 110.06 last 169.69) - open/close 20000 (time 1746067713.42 total 119.31 last 164.18) - open/close 21870 (time 1746067723.43 total 129.31 last 186.94) - open/close 23823 (time 1746067733.43 total 139.31 last 195.28) - open/close 25710 (time 1746067743.43 total 149.31 last 188.69) - open/close 27413 (time 1746067753.44 total 159.32 last 170.19) - open/close 29124 (time 1746067763.44 total 169.33 last 171.00) - open/close 30000 (time 1746067767.74 total 173.62 last 203.93) - open/close 31953 (time 1746067777.74 total 183.62 last 195.29) - open/close 33790 (time 1746067787.74 total 193.62 last 183.69) - open/close 35751 (time 1746067797.74 total 203.62 last 196.08) - open/close 38164 (time 1746067807.74 total 213.62 last 241.28) - open/close 40000 (time 1746067816.65 total 222.53 last 206.14) - open/close 42043 (time 1746067826.65 total 232.54 last 204.17) - open/close 43724 (time 1746067836.66 total 242.54 last 167.97) - open/close 45620 (time 1746067846.67 total 252.55 last 189.41) - open/close 47021 (time 1746067856.67 total 262.56 last 140.04) - open/close 48641 (time 1746067866.68 total 272.56 last 161.92) - open/close 50000 (time 1746067874.97 total 280.85 last 163.90) - open/close 51653 (time 1746067884.98 total 290.87 last 165.13) - open/close 53316 (time 1746067894.99 total 300.87 last 166.22) - open/close 54983 (time 1746067904.99 total 310.87 last 166.68) - open/close 56545 (time 1746067914.99 total 320.87 last 156.16) - open/close 58064 (time 1746067924.99 total 330.88 last 151.87) - open/close 59447 (time 1746067935.00 total 340.88 last 138.21) - open/close 60000 (time 1746067938.89 total 344.78 last 142.03) - open/close 61424 (time 1746067948.90 total 354.78 last 142.34) - open/close 62984 (time 1746067958.90 total 364.78 last 155.93) - open/close 64703 (time 1746067968.91 total 374.79 last 171.79) total: 64768 open/close in 375.20 seconds: 172.62 ops/second - /unlink 2771 (time 1746067980.73 total 10.00 last 277.08) - /unlink 5825 (time 1746067990.73 total 20.00 last 305.37) - /unlink 8246 (time 1746068000.73 total 30.00 last 242.03) - /unlink 10000 (time 1746068007.96 total 37.24 last 242.43) - /unlink 12945 (time 1746068017.97 total 47.24 last 294.46) - /unlink 16920 (time 1746068027.97 total 57.24 last 397.43) - /unlink 20000 (time 1746068037.80 total 67.08 last 313.13) - /unlink 22415 (time 1746068047.81 total 77.08 last 241.42) - /unlink 25095 (time 1746068057.81 total 87.08 last 268.00) - /unlink 27725 (time 1746068067.81 total 97.08 last 262.95) - /unlink 30000 (time 1746068076.15 total 105.42 last 272.87) - /unlink 32796 (time 1746068086.15 total 115.42 last 279.59) - /unlink 35991 (time 1746068096.15 total 125.42 last 319.44) - /unlink 39062 (time 1746068106.15 total 135.43 last 307.06) - /unlink 40000 (time 1746068110.00 total 139.28 last 243.49) - /unlink 42448 (time 1746068120.00 total 149.28 last 244.78) - /unlink 45154 (time 1746068130.00 total 159.28 last 270.56) - /unlink 47761 (time 1746068140.01 total 169.28 last 260.60) - /unlink 50000 (time 1746068148.47 total 177.75 last 264.49) - /unlink 53325 (time 1746068158.47 total 187.75 last 332.49) - /unlink 56639 (time 1746068168.48 total 197.75 last 331.33) - /unlink 60000 (time 1746068176.34 total 205.62 last 427.18) - /unlink 63430 (time 1746068186.35 total 215.62 last 342.92) total: 64768 /unlink in 220.22 seconds: 294.10 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1606 (time 1746068220.20 total 10.01 last 160.49) - open/close 3222 (time 1746068230.20 total 20.01 last 161.57) - open/close 4656 (time 1746068240.21 total 30.01 last 143.32) - open/close 6768 (time 1746068250.21 total 40.02 last 211.18) - open/close 8731 (time 1746068260.22 total 50.02 last 196.16) - open/close 10000 (time 1746068267.54 total 57.34 last 173.38) - open/close 11867 (time 1746068277.54 total 67.34 last 186.65) - open/close 13570 (time 1746068287.54 total 77.35 last 170.26) - open/close 15157 (time 1746068297.54 total 87.35 last 158.70) - open/close 16615 (time 1746068307.55 total 97.35 last 145.75) - open/close 17834 (time 1746068317.55 total 107.35 last 121.86) - open/close 19127 (time 1746068327.57 total 117.37 last 129.07) - open/close 20000 (time 1746068333.25 total 123.06 last 153.51) - open/close 21517 (time 1746068343.25 total 133.06 last 151.69) - open/close 23513 (time 1746068353.26 total 143.06 last 199.54) - open/close 25590 (time 1746068363.26 total 153.06 last 207.68) - open/close 28126 (time 1746068373.26 total 163.06 last 253.57) - open/close 30000 (time 1746068380.14 total 169.94 last 272.48) - open/close 32799 (time 1746068390.14 total 179.94 last 279.82) - open/close 35175 (time 1746068400.14 total 189.95 last 237.59) - open/close 37371 (time 1746068410.14 total 199.95 last 219.58) - open/close 39623 (time 1746068420.14 total 209.95 last 225.15) - open/close 40000 (time 1746068422.49 total 212.30 last 160.61) - open/close 41913 (time 1746068432.49 total 222.30 last 191.29) - open/close 43756 (time 1746068442.50 total 232.30 last 184.23) - open/close 45597 (time 1746068452.50 total 242.30 last 184.08) - open/close 47278 (time 1746068462.50 total 252.30 last 168.08) - open/close 49394 (time 1746068472.50 total 262.30 last 211.58) - open/close 50000 (time 1746068475.68 total 265.49 last 190.19) - open/close 52313 (time 1746068485.69 total 275.49 last 231.29) - open/close 54170 (time 1746068495.69 total 285.49 last 185.63) - open/close 56571 (time 1746068505.69 total 295.49 last 240.09) - open/close 58985 (time 1746068515.69 total 305.50 last 241.35) - open/close 60000 (time 1746068520.23 total 310.03 last 223.71) - open/close 62670 (time 1746068530.23 total 320.04 last 266.86) total: 64768 open/close in 328.37 seconds: 197.24 ops/second - /unlink 4289 (time 1746068549.64 total 10.00 last 428.85) - /unlink 8310 (time 1746068559.64 total 20.00 last 402.09) - /unlink 10000 (time 1746068565.15 total 25.51 last 306.57) - /unlink 13226 (time 1746068575.15 total 35.52 last 322.57) - /unlink 16292 (time 1746068585.15 total 45.52 last 306.51) - /unlink 19037 (time 1746068595.15 total 55.52 last 274.47) - /unlink 20000 (time 1746068598.78 total 59.15 last 265.46) - /unlink 22826 (time 1746068608.78 total 69.15 last 282.59) - /unlink 26121 (time 1746068618.78 total 79.15 last 329.49) - /unlink 29096 (time 1746068628.78 total 89.15 last 297.47) - /unlink 30000 (time 1746068631.76 total 92.12 last 303.72) - /unlink 33707 (time 1746068641.76 total 102.12 last 370.69) - /unlink 36890 (time 1746068651.76 total 112.13 last 318.26) - /unlink 40000 (time 1746068661.22 total 121.58 last 328.82) - /unlink 42887 (time 1746068671.22 total 131.59 last 288.67) - /unlink 45109 (time 1746068681.22 total 141.59 last 222.19) - /unlink 47155 (time 1746068691.22 total 151.59 last 204.56) - /unlink 50000 (time 1746068700.57 total 160.93 last 304.41) - /unlink 53088 (time 1746068710.57 total 170.94 last 308.74) - /unlink 57057 (time 1746068720.57 total 180.94 last 396.84) - /unlink 60000 (time 1746068726.90 total 187.27 last 464.94) - /unlink 63431 (time 1746068736.90 total 197.27 last 343.03) total: 64768 /unlink in 200.66 seconds: 322.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1762 (time 1746068769.38 total 10.01 last 176.09) - open/close 3110 (time 1746068779.39 total 20.01 last 134.72) - open/close 4569 (time 1746068789.40 total 30.02 last 145.84) - open/close 6255 (time 1746068799.40 total 40.02 last 168.52) - open/close 7888 (time 1746068809.40 total 50.02 last 163.29) - open/close 9544 (time 1746068819.41 total 60.03 last 165.45) - open/close 10000 (time 1746068821.91 total 62.53 last 182.66) - open/close 11993 (time 1746068831.91 total 72.53 last 199.26) - open/close 14427 (time 1746068841.91 total 82.54 last 243.27) - open/close 16531 (time 1746068851.92 total 92.54 last 210.36) - open/close 18626 (time 1746068861.92 total 102.54 last 209.44) - open/close 20000 (time 1746068870.02 total 110.64 last 169.63) - open/close 21704 (time 1746068880.02 total 120.65 last 170.30) - open/close 23263 (time 1746068890.02 total 130.65 last 155.89) - open/close 25162 (time 1746068900.03 total 140.65 last 189.74) - open/close 26789 (time 1746068910.03 total 150.65 last 162.69) - open/close 28534 (time 1746068920.03 total 160.66 last 174.48) - open/close 30000 (time 1746068926.55 total 167.17 last 224.90) - open/close 31792 (time 1746068936.55 total 177.17 last 179.19) - open/close 33475 (time 1746068946.56 total 187.18 last 168.20) - open/close 35033 (time 1746068956.56 total 197.18 last 155.74) - open/close 36646 (time 1746068966.57 total 207.19 last 161.22) - open/close 38412 (time 1746068976.57 total 217.19 last 176.57) - open/close 40000 (time 1746068986.12 total 226.74 last 166.35) - open/close 42326 (time 1746068996.12 total 236.74 last 232.55) - open/close 44091 (time 1746069006.12 total 246.74 last 176.45) - open/close 45774 (time 1746069016.13 total 256.75 last 168.21) - open/close 47498 (time 1746069026.13 total 266.75 last 172.39) - open/close 49130 (time 1746069036.13 total 276.75 last 163.14) - open/close 50000 (time 1746069040.52 total 281.14 last 198.07) - open/close 51789 (time 1746069050.53 total 291.15 last 178.85) - open/close 53508 (time 1746069060.53 total 301.15 last 171.85) - open/close 55364 (time 1746069070.53 total 311.15 last 185.55) - open/close 56954 (time 1746069080.53 total 321.16 last 158.95) - open/close 58355 (time 1746069090.54 total 331.16 last 140.00) - open/close 59932 (time 1746069100.54 total 341.16 last 157.70) - open/close 61347 (time 1746069110.55 total 351.17 last 141.43) - open/close 62696 (time 1746069120.55 total 361.17 last 134.88) - open/close 63970 (time 1746069130.55 total 371.17 last 127.36) total: 64768 open/close in 376.97 seconds: 171.81 ops/second - /unlink 2336 (time 1746069147.93 total 10.00 last 233.60) - /unlink 4541 (time 1746069157.93 total 20.00 last 220.45) - /unlink 6430 (time 1746069167.94 total 30.01 last 188.79) - /unlink 8662 (time 1746069177.94 total 40.01 last 223.19) - /unlink 10000 (time 1746069184.16 total 46.23 last 215.14) - /unlink 12232 (time 1746069194.16 total 56.23 last 223.12) - /unlink 14667 (time 1746069204.16 total 66.23 last 243.48) - /unlink 17439 (time 1746069214.17 total 76.24 last 277.12) - /unlink 19716 (time 1746069224.17 total 86.24 last 227.63) - /unlink 22094 (time 1746069234.17 total 96.24 last 237.74) - /unlink 24403 (time 1746069244.18 total 106.24 last 230.84) - /unlink 27178 (time 1746069254.18 total 116.25 last 277.41) - /unlink 29803 (time 1746069264.18 total 126.25 last 262.48) - /unlink 32419 (time 1746069274.18 total 136.25 last 261.59) - /unlink 34951 (time 1746069284.18 total 146.25 last 253.08) - /unlink 37468 (time 1746069294.19 total 156.26 last 251.61) - /unlink 39568 (time 1746069304.19 total 166.26 last 209.93) - /unlink 40000 (time 1746069306.42 total 168.49 last 193.87) - /unlink 41981 (time 1746069316.42 total 178.49 last 198.08) - /unlink 44033 (time 1746069326.42 total 188.49 last 205.18) - /unlink 46380 (time 1746069336.43 total 198.50 last 234.54) - /unlink 49193 (time 1746069346.43 total 208.50 last 281.29) - /unlink 50000 (time 1746069349.61 total 211.68 last 253.86) - /unlink 52196 (time 1746069359.61 total 221.68 last 219.56) - /unlink 54380 (time 1746069369.61 total 231.68 last 218.31) - /unlink 56809 (time 1746069379.61 total 241.68 last 242.90) - /unlink 59499 (time 1746069389.61 total 251.68 last 268.99) - /unlink 60000 (time 1746069391.65 total 253.72 last 245.79) - /unlink 62521 (time 1746069401.66 total 263.72 last 252.00) total: 64768 /unlink in 272.59 seconds: 237.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server unloading modules via unload_modules_local on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 PASS 106 (3462s) Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg250-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3567 sec ========== 23:17:53 (1746069473) === conf-sanity: start cleanup 23:17:54 (1746069474) === === conf-sanity: finish cleanup 23:17:55 (1746069475) ===