-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 23:42:15 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg416-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg416-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg416-server oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg416-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg416-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg416-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg416-server: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg416-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg416-server: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg416-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:44:06 (1746071046) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions oleg416-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg416-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg416-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg416-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1624 (time 1746071102.18 total 10.00 last 162.38) - open/close 3060 (time 1746071112.18 total 20.00 last 143.59) - open/close 4633 (time 1746071122.18 total 30.00 last 157.28) - open/close 6332 (time 1746071132.19 total 40.01 last 169.79) - open/close 8360 (time 1746071142.19 total 50.01 last 202.79) - open/close 10000 (time 1746071148.64 total 56.46 last 254.06) - open/close 12625 (time 1746071158.65 total 66.47 last 262.44) - open/close 15214 (time 1746071168.65 total 76.47 last 258.79) - open/close 17093 (time 1746071178.65 total 86.47 last 187.90) - open/close 19170 (time 1746071188.66 total 96.48 last 207.56) - open/close 20000 (time 1746071192.19 total 100.01 last 235.14) - open/close 22031 (time 1746071202.19 total 110.01 last 203.01) - open/close 24389 (time 1746071212.19 total 120.01 last 235.77) - open/close 26591 (time 1746071222.20 total 130.02 last 220.13) - open/close 28258 (time 1746071232.20 total 140.02 last 166.63) - open/close 29904 (time 1746071242.20 total 150.02 last 164.54) - open/close 31282 (time 1746071252.21 total 160.03 last 137.77) - open/close 32523 (time 1746071262.22 total 170.04 last 123.92) - open/close 33906 (time 1746071272.23 total 180.05 last 138.22) - open/close 35739 (time 1746071282.23 total 190.05 last 183.26) - open/close 37057 (time 1746071292.23 total 200.06 last 131.70) - open/close 38378 (time 1746071302.24 total 210.06 last 132.04) - open/close 39806 (time 1746071312.24 total 220.06 last 142.80) - open/close 41201 (time 1746071322.24 total 230.06 last 139.48) - open/close 42729 (time 1746071332.24 total 240.06 last 152.79) - open/close 44276 (time 1746071342.24 total 250.07 last 154.66) - open/close 46090 (time 1746071352.25 total 260.07 last 181.34) - open/close 47862 (time 1746071362.25 total 270.07 last 177.12) - open/close 49275 (time 1746071372.26 total 280.08 last 141.26) - open/close 50000 (time 1746071376.91 total 284.73 last 155.75) - open/close 51422 (time 1746071386.91 total 294.73 last 142.17) - open/close 52901 (time 1746071396.92 total 304.74 last 147.82) - open/close 54300 (time 1746071406.92 total 314.74 last 139.85) - open/close 56047 (time 1746071416.93 total 324.75 last 174.60) - open/close 57627 (time 1746071426.93 total 334.76 last 157.90) - open/close 59745 (time 1746071436.93 total 344.76 last 211.78) - open/close 61549 (time 1746071446.94 total 354.76 last 180.29) - open/close 63292 (time 1746071456.95 total 364.77 last 174.14) total: 64768 open/close in 372.07 seconds: 174.08 ops/second - /unlink 3015 (time 1746071475.72 total 10.01 last 301.35) - /unlink 6402 (time 1746071485.73 total 20.01 last 338.59) - /unlink 10000 (time 1746071494.98 total 29.26 last 388.71) - /unlink 12962 (time 1746071504.98 total 39.26 last 296.19) - /unlink 15604 (time 1746071514.99 total 49.27 last 264.06) - /unlink 17838 (time 1746071524.99 total 59.27 last 223.30) - /unlink 20000 (time 1746071534.40 total 68.68 last 229.81) - /unlink 22501 (time 1746071544.41 total 78.69 last 249.96) - /unlink 24700 (time 1746071554.41 total 88.69 last 219.85) - /unlink 27202 (time 1746071564.41 total 98.69 last 250.11) - /unlink 29545 (time 1746071574.42 total 108.70 last 234.08) - /unlink 30000 (time 1746071576.69 total 110.97 last 200.48) - /unlink 32702 (time 1746071586.69 total 120.97 last 270.16) - /unlink 35375 (time 1746071596.69 total 130.98 last 267.26) - /unlink 38026 (time 1746071606.70 total 140.98 last 265.05) - /unlink 40000 (time 1746071613.14 total 147.43 last 306.17) - /unlink 42386 (time 1746071623.14 total 157.43 last 238.58) - /unlink 44575 (time 1746071633.14 total 167.43 last 218.90) - /unlink 46941 (time 1746071643.14 total 177.43 last 236.60) - /unlink 49526 (time 1746071653.14 total 187.43 last 258.47) - /unlink 52206 (time 1746071663.15 total 197.43 last 267.93) - /unlink 54859 (time 1746071673.15 total 207.43 last 265.28) - /unlink 57477 (time 1746071683.15 total 217.43 last 261.74) - /unlink 59871 (time 1746071693.15 total 227.43 last 239.39) - /unlink 63002 (time 1746071703.15 total 237.43 last 313.06) total: 64768 /unlink in 242.30 seconds: 267.31 ops/second Waiting for MDT destroys to complete - open/close 1833 (time 1746071729.42 total 10.00 last 183.24) - open/close 3461 (time 1746071739.43 total 20.00 last 162.77) - open/close 5179 (time 1746071749.43 total 30.01 last 171.75) - open/close 6929 (time 1746071759.43 total 40.01 last 174.91) - open/close 8619 (time 1746071769.44 total 50.02 last 168.91) - open/close 9915 (time 1746071779.44 total 60.02 last 129.60) - open/close 11320 (time 1746071789.44 total 70.02 last 140.45) - open/close 12835 (time 1746071799.45 total 80.03 last 151.39) - open/close 14821 (time 1746071809.45 total 90.03 last 198.57) - open/close 16561 (time 1746071819.45 total 100.03 last 173.97) - open/close 18834 (time 1746071829.45 total 110.03 last 227.30) - open/close 20000 (time 1746071833.90 total 114.48 last 262.38) - open/close 21898 (time 1746071843.90 total 124.48 last 189.68) - open/close 24184 (time 1746071853.90 total 134.48 last 228.60) - open/close 26426 (time 1746071863.91 total 144.49 last 224.15) - open/close 28084 (time 1746071873.91 total 154.49 last 165.76) - open/close 29510 (time 1746071883.91 total 164.49 last 142.59) - open/close 30000 (time 1746071887.35 total 167.93 last 142.43) - open/close 31338 (time 1746071897.35 total 177.93 last 133.79) - open/close 33432 (time 1746071907.35 total 187.93 last 209.40) - open/close 35117 (time 1746071917.35 total 197.93 last 168.44) - open/close 36951 (time 1746071927.36 total 207.94 last 183.35) - open/close 38857 (time 1746071937.36 total 217.94 last 190.44) - open/close 40000 (time 1746071943.98 total 224.56 last 172.83) - open/close 41678 (time 1746071953.98 total 234.56 last 167.78) - open/close 43731 (time 1746071963.98 total 244.56 last 205.29) - open/close 45876 (time 1746071973.99 total 254.57 last 214.31) - open/close 47989 (time 1746071983.99 total 264.57 last 211.27) - open/close 50000 (time 1746071992.03 total 272.61 last 250.15) - open/close 52211 (time 1746072002.03 total 282.61 last 221.02) - open/close 54586 (time 1746072012.04 total 292.62 last 237.43) - open/close 56951 (time 1746072022.04 total 302.62 last 236.44) - open/close 58809 (time 1746072032.04 total 312.62 last 185.79) - open/close 60000 (time 1746072038.60 total 319.18 last 181.59) - open/close 62106 (time 1746072048.60 total 329.18 last 210.46) - open/close 64049 (time 1746072058.61 total 339.19 last 194.23) total: 64768 open/close in 342.29 seconds: 189.22 ops/second - /unlink 4520 (time 1746072072.95 total 10.00 last 451.80) - /unlink 9159 (time 1746072082.95 total 20.00 last 463.89) - /unlink 10000 (time 1746072084.97 total 22.03 last 415.61) - /unlink 15287 (time 1746072094.98 total 32.03 last 528.50) - /unlink 19235 (time 1746072104.98 total 42.03 last 394.79) - /unlink 24401 (time 1746072114.98 total 52.03 last 516.50) - /unlink 29162 (time 1746072124.98 total 62.03 last 476.09) - /unlink 30000 (time 1746072127.19 total 64.25 last 378.34) - /unlink 33947 (time 1746072137.20 total 74.25 last 394.66) - /unlink 37157 (time 1746072147.20 total 84.25 last 320.94) - /unlink 39866 (time 1746072157.20 total 94.25 last 270.85) - /unlink 43091 (time 1746072167.20 total 104.25 last 322.48) - /unlink 46163 (time 1746072177.20 total 114.25 last 307.20) - /unlink 48845 (time 1746072187.20 total 124.26 last 268.09) - /unlink 50000 (time 1746072190.50 total 127.55 last 350.58) - /unlink 53581 (time 1746072200.50 total 137.56 last 358.00) - /unlink 56830 (time 1746072210.50 total 147.56 last 324.89) - /unlink 59728 (time 1746072220.51 total 157.56 last 289.65) - /unlink 62534 (time 1746072230.51 total 167.56 last 280.57) total: 64768 /unlink in 173.93 seconds: 372.38 ops/second Waiting for MDT destroys to complete - open/close 1676 (time 1746072254.72 total 10.00 last 167.54) - open/close 3328 (time 1746072264.72 total 20.01 last 165.14) - open/close 5235 (time 1746072274.73 total 30.01 last 190.64) - open/close 7545 (time 1746072284.73 total 40.01 last 230.90) - open/close 9286 (time 1746072294.73 total 50.02 last 174.07) - open/close 10000 (time 1746072298.35 total 53.64 last 197.08) - open/close 11710 (time 1746072308.36 total 63.64 last 170.91) - open/close 13002 (time 1746072318.36 total 73.65 last 129.18) - open/close 14509 (time 1746072328.36 total 83.65 last 150.70) - open/close 16037 (time 1746072338.36 total 93.65 last 152.79) - open/close 17760 (time 1746072348.36 total 103.65 last 172.26) - open/close 19251 (time 1746072358.37 total 113.65 last 149.03) - open/close 20000 (time 1746072363.50 total 118.79 last 145.86) - open/close 21811 (time 1746072373.51 total 128.80 last 180.97) - open/close 23518 (time 1746072383.51 total 138.80 last 170.69) - open/close 25054 (time 1746072393.52 total 148.80 last 153.52) - open/close 26859 (time 1746072403.52 total 158.80 last 180.48) - open/close 28452 (time 1746072413.52 total 168.81 last 159.25) - open/close 29916 (time 1746072423.52 total 178.81 last 146.39) - open/close 31537 (time 1746072433.53 total 188.81 last 162.06) - open/close 33114 (time 1746072443.53 total 198.82 last 157.62) - open/close 34623 (time 1746072453.53 total 208.82 last 150.88) - open/close 36486 (time 1746072463.53 total 218.82 last 186.29) - open/close 38007 (time 1746072473.54 total 228.82 last 151.98) - open/close 39611 (time 1746072483.54 total 238.82 last 160.39) - open/close 40000 (time 1746072486.09 total 241.37 last 152.86) - open/close 41537 (time 1746072496.09 total 251.37 last 153.67) - open/close 43098 (time 1746072506.09 total 261.38 last 156.00) - open/close 44764 (time 1746072516.10 total 271.38 last 166.55) - open/close 46527 (time 1746072526.10 total 281.39 last 176.17) - open/close 48048 (time 1746072536.11 total 291.40 last 152.00) - open/close 49499 (time 1746072546.12 total 301.40 last 145.03) - open/close 50000 (time 1746072549.96 total 305.24 last 130.49) - open/close 51628 (time 1746072559.96 total 315.24 last 162.79) - open/close 53259 (time 1746072569.96 total 325.24 last 163.05) - open/close 54579 (time 1746072579.97 total 335.25 last 131.92) - open/close 55984 (time 1746072589.97 total 345.25 last 140.45) - open/close 57752 (time 1746072599.97 total 355.26 last 176.72) - open/close 59753 (time 1746072609.98 total 365.26 last 200.04) - open/close 61426 (time 1746072619.98 total 375.26 last 167.27) - open/close 63254 (time 1746072629.98 total 385.27 last 182.74) total: 64768 open/close in 391.44 seconds: 165.46 ops/second - /unlink 3123 (time 1746072647.28 total 10.00 last 312.22) - /unlink 6168 (time 1746072657.28 total 20.00 last 304.43) - /unlink 9105 (time 1746072667.28 total 30.01 last 293.60) - /unlink 10000 (time 1746072670.61 total 33.34 last 268.77) - /unlink 13226 (time 1746072680.61 total 43.34 last 322.56) - /unlink 16425 (time 1746072690.62 total 53.34 last 319.74) - /unlink 18662 (time 1746072700.62 total 63.35 last 223.66) - /unlink 20000 (time 1746072706.27 total 68.99 last 236.95) - /unlink 22729 (time 1746072716.27 total 79.00 last 272.72) - /unlink 25793 (time 1746072726.28 total 89.00 last 306.33) - /unlink 28501 (time 1746072736.28 total 99.00 last 270.78) - /unlink 30000 (time 1746072742.23 total 104.95 last 252.03) - /unlink 32659 (time 1746072752.23 total 114.95 last 265.89) - /unlink 34909 (time 1746072762.23 total 124.95 last 224.93) - /unlink 37237 (time 1746072772.23 total 134.96 last 232.73) - /unlink 40000 (time 1746072781.23 total 143.95 last 307.21) - /unlink 42582 (time 1746072791.23 total 153.95 last 258.18) - /unlink 44942 (time 1746072801.23 total 163.95 last 235.95) - /unlink 47342 (time 1746072811.23 total 173.96 last 239.96) - /unlink 49762 (time 1746072821.23 total 183.96 last 241.94) - /unlink 51770 (time 1746072831.24 total 193.96 last 200.75) - /unlink 53882 (time 1746072841.24 total 203.97 last 211.07) - /unlink 56016 (time 1746072851.24 total 213.97 last 213.36) - /unlink 58487 (time 1746072861.25 total 223.97 last 246.99) - /unlink 60000 (time 1746072869.12 total 231.84 last 192.24) - /unlink 62098 (time 1746072879.13 total 241.85 last 209.62) - /unlink 64425 (time 1746072889.13 total 251.85 last 232.64) total: 64768 /unlink in 253.25 seconds: 255.75 ops/second Waiting for MDT destroys to complete - open/close 1357 (time 1746072912.27 total 10.01 last 135.58) - open/close 2733 (time 1746072922.27 total 20.01 last 137.58) - open/close 4090 (time 1746072932.28 total 30.02 last 135.61) - open/close 5302 (time 1746072942.29 total 40.03 last 121.04) - open/close 6525 (time 1746072952.29 total 50.03 last 122.26) - open/close 7795 (time 1746072962.30 total 60.04 last 126.88) - open/close 9048 (time 1746072972.31 total 70.05 last 125.25) - open/close 10000 (time 1746072979.43 total 77.17 last 133.65) - open/close 11385 (time 1746072989.43 total 87.17 last 138.49) - open/close 12757 (time 1746072999.44 total 97.18 last 137.15) - open/close 14270 (time 1746073009.44 total 107.18 last 151.29) - open/close 15602 (time 1746073019.44 total 117.18 last 133.17) - open/close 16973 (time 1746073029.44 total 127.18 last 137.03) - open/close 18337 (time 1746073039.44 total 137.18 last 136.40) - open/close 19854 (time 1746073049.45 total 147.19 last 151.62) - open/close 21402 (time 1746073059.45 total 157.19 last 154.77) - open/close 22902 (time 1746073069.49 total 167.23 last 149.45) - open/close 24326 (time 1746073079.49 total 177.23 last 142.36) - open/close 25691 (time 1746073089.51 total 187.25 last 136.29) - open/close 26924 (time 1746073099.51 total 197.25 last 123.26) - open/close 28431 (time 1746073109.51 total 207.25 last 150.63) - open/close 29836 (time 1746073119.52 total 217.26 last 140.45) - open/close 31162 (time 1746073129.52 total 227.26 last 132.54) - open/close 32453 (time 1746073139.52 total 237.26 last 129.09) - open/close 33758 (time 1746073149.53 total 247.27 last 130.43) - open/close 35267 (time 1746073159.54 total 257.28 last 150.75) - open/close 36481 (time 1746073169.55 total 267.29 last 121.28) - open/close 37802 (time 1746073179.56 total 277.30 last 131.98) - open/close 39026 (time 1746073189.56 total 287.30 last 122.32) - open/close 40000 (time 1746073197.82 total 295.56 last 118.01) - open/close 41351 (time 1746073207.82 total 305.56 last 135.05) - open/close 42398 (time 1746073217.83 total 315.57 last 104.63) - open/close 43537 (time 1746073227.83 total 325.57 last 113.83) - open/close 44707 (time 1746073237.84 total 335.58 last 116.97) - open/close 45831 (time 1746073247.84 total 345.58 last 112.35) - open/close 47020 (time 1746073257.84 total 355.58 last 118.86) - open/close 48133 (time 1746073267.85 total 365.59 last 111.27) - open/close 49256 (time 1746073277.85 total 375.59 last 112.30) - open/close 50000 (time 1746073283.69 total 381.43 last 127.22) - open/close 51077 (time 1746073293.70 total 391.44 last 107.63) - open/close 52174 (time 1746073303.70 total 401.44 last 109.68) - open/close 53396 (time 1746073313.71 total 411.45 last 122.14) - open/close 54521 (time 1746073323.71 total 421.45 last 112.44) - open/close 55610 (time 1746073333.73 total 431.47 last 108.76) - open/close 56817 (time 1746073343.73 total 441.47 last 120.67) - open/close 58203 (time 1746073353.73 total 451.47 last 138.54) - open/close 59666 (time 1746073363.74 total 461.48 last 146.22) - open/close 60000 (time 1746073366.21 total 463.95 last 134.88) - open/close 61385 (time 1746073376.21 total 473.95 last 138.50) - open/close 62555 (time 1746073386.23 total 483.97 last 116.80) - open/close 63827 (time 1746073396.24 total 493.98 last 127.10) total: 64768 open/close in 500.95 seconds: 129.29 ops/second - /unlink 2660 (time 1746073415.18 total 10.00 last 265.87) - /unlink 5039 (time 1746073425.18 total 20.01 last 237.85) - /unlink 7430 (time 1746073435.19 total 30.01 last 239.05) - /unlink 9443 (time 1746073445.19 total 40.01 last 201.28) - /unlink 10000 (time 1746073447.93 total 42.75 last 203.10) - /unlink 12013 (time 1746073457.94 total 52.76 last 201.16) - /unlink 13982 (time 1746073467.94 total 62.76 last 196.80) - /unlink 16143 (time 1746073477.94 total 72.76 last 216.10) - /unlink 18303 (time 1746073487.94 total 82.77 last 215.96) - /unlink 20000 (time 1746073496.12 total 90.94 last 207.67) - /unlink 22409 (time 1746073506.12 total 100.94 last 240.83) - /unlink 24943 (time 1746073516.13 total 110.95 last 253.22) - /unlink 27628 (time 1746073526.14 total 120.96 last 268.22) - /unlink 29793 (time 1746073536.14 total 130.96 last 216.50) - /unlink 31890 (time 1746073546.16 total 140.98 last 209.28) - /unlink 33881 (time 1746073556.16 total 150.98 last 199.05) - /unlink 36051 (time 1746073566.17 total 160.99 last 216.85) - /unlink 38304 (time 1746073576.17 total 170.99 last 225.26) - /unlink 40000 (time 1746073584.90 total 179.72 last 194.21) - /unlink 42090 (time 1746073594.90 total 189.72 last 208.99) - /unlink 44258 (time 1746073604.91 total 199.74 last 216.50) - /unlink 46375 (time 1746073614.92 total 209.74 last 211.54) - /unlink 48469 (time 1746073624.92 total 219.75 last 209.39) - /unlink 50000 (time 1746073632.09 total 226.91 last 213.56) - /unlink 52231 (time 1746073642.10 total 236.92 last 222.90) - /unlink 54304 (time 1746073652.10 total 246.93 last 207.22) - /unlink 56360 (time 1746073662.11 total 256.93 last 205.51) - /unlink 58548 (time 1746073672.11 total 266.93 last 218.77) - /unlink 60000 (time 1746073678.35 total 273.18 last 232.58) - /unlink 62262 (time 1746073688.35 total 283.18 last 226.18) - /unlink 64649 (time 1746073698.36 total 293.18 last 238.68) total: 64768 /unlink in 293.70 seconds: 220.52 ops/second Waiting for MDT destroys to complete - open/close 1498 (time 1746073718.45 total 10.00 last 149.77) - open/close 2814 (time 1746073728.45 total 20.00 last 131.57) - open/close 4045 (time 1746073738.46 total 30.01 last 123.09) - open/close 5300 (time 1746073748.46 total 40.01 last 125.46) - open/close 6725 (time 1746073758.47 total 50.02 last 142.40) - open/close 8003 (time 1746073768.47 total 60.02 last 127.75) - open/close 9275 (time 1746073778.48 total 70.02 last 127.13) - open/close 10000 (time 1746073784.30 total 75.85 last 124.50) - open/close 11332 (time 1746073794.30 total 85.85 last 133.19) - open/close 12714 (time 1746073804.30 total 95.85 last 138.19) - open/close 14011 (time 1746073814.31 total 105.85 last 129.63) - open/close 15450 (time 1746073824.31 total 115.86 last 143.87) - open/close 16863 (time 1746073834.31 total 125.86 last 141.21) - open/close 18450 (time 1746073844.32 total 135.87 last 158.62) - open/close 19993 (time 1746073854.32 total 145.87 last 154.23) - open/close 21573 (time 1746073864.33 total 155.88 last 157.91) - open/close 23118 (time 1746073874.33 total 165.88 last 154.47) - open/close 24468 (time 1746073884.34 total 175.89 last 134.90) - open/close 26105 (time 1746073894.35 total 185.90 last 163.47) - open/close 27651 (time 1746073904.35 total 195.90 last 154.57) - open/close 29213 (time 1746073914.36 total 205.91 last 156.15) - open/close 30000 (time 1746073918.72 total 210.27 last 180.26) - open/close 31726 (time 1746073928.73 total 220.28 last 172.56) - open/close 33589 (time 1746073938.73 total 230.28 last 186.22) - open/close 35208 (time 1746073948.74 total 240.29 last 161.76) - open/close 36706 (time 1746073958.75 total 250.30 last 149.63) - open/close 38180 (time 1746073968.75 total 260.30 last 147.36) - open/close 39733 (time 1746073978.76 total 270.31 last 155.18) - open/close 41144 (time 1746073988.77 total 280.32 last 141.03) - open/close 42519 (time 1746073998.77 total 290.32 last 137.46) - open/close 43912 (time 1746074008.77 total 300.32 last 139.24) - open/close 45369 (time 1746074018.78 total 310.33 last 145.66) - open/close 46864 (time 1746074028.78 total 320.33 last 149.44) - open/close 48530 (time 1746074038.78 total 330.33 last 166.57) - open/close 50000 (time 1746074046.90 total 338.45 last 180.99) - open/close 51541 (time 1746074056.90 total 348.45 last 154.09) - open/close 52919 (time 1746074066.92 total 358.46 last 137.65) - open/close 54406 (time 1746074076.92 total 368.47 last 148.65) - open/close 56113 (time 1746074086.92 total 378.47 last 170.60) - open/close 57776 (time 1746074096.93 total 388.47 last 166.28) - open/close 59522 (time 1746074106.93 total 398.48 last 174.55) - open/close 60000 (time 1746074110.50 total 402.05 last 133.99) - open/close 61510 (time 1746074120.50 total 412.05 last 150.97) - open/close 63099 (time 1746074130.50 total 422.05 last 158.81) - open/close 64529 (time 1746074140.51 total 432.06 last 142.95) total: 64768 open/close in 433.59 seconds: 149.38 ops/second - /unlink 2857 (time 1746074153.73 total 10.00 last 285.60) - /unlink 5013 (time 1746074163.74 total 20.01 last 215.54) - /unlink 7288 (time 1746074173.74 total 30.01 last 227.34) - /unlink 10000 (time 1746074183.02 total 39.28 last 292.50) - /unlink 12795 (time 1746074193.02 total 49.29 last 279.47) - /unlink 16661 (time 1746074203.02 total 59.29 last 386.58) - /unlink 19647 (time 1746074213.02 total 69.29 last 298.58) - /unlink 21952 (time 1746074223.02 total 79.29 last 230.47) - /unlink 24507 (time 1746074233.02 total 89.29 last 255.44) - /unlink 27265 (time 1746074243.02 total 99.29 last 275.78) - /unlink 30000 (time 1746074252.16 total 108.43 last 299.20) - /unlink 32571 (time 1746074262.17 total 118.43 last 257.06) - /unlink 34880 (time 1746074272.17 total 128.43 last 230.89) - /unlink 37152 (time 1746074282.17 total 138.44 last 227.16) - /unlink 39459 (time 1746074292.17 total 148.44 last 230.61) - /unlink 40000 (time 1746074294.64 total 150.90 last 219.54) - /unlink 42112 (time 1746074304.64 total 160.91 last 211.00) - /unlink 44281 (time 1746074314.65 total 170.92 last 216.83) - /unlink 46508 (time 1746074324.65 total 180.92 last 222.56) - /unlink 48800 (time 1746074334.66 total 190.93 last 229.13) - /unlink 50000 (time 1746074339.07 total 195.34 last 271.92) - /unlink 52562 (time 1746074349.07 total 205.34 last 256.14) - /unlink 55036 (time 1746074359.07 total 215.34 last 247.38) - /unlink 57665 (time 1746074369.08 total 225.35 last 262.79) - /unlink 60000 (time 1746074377.35 total 233.62 last 282.16) - /unlink 63145 (time 1746074387.35 total 243.62 last 314.45) total: 64768 /unlink in 248.48 seconds: 260.66 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg416-server unloading modules via unload_modules_local on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 PASS 106 (3380s) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions oleg416-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg416-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3519 sec ========== 00:40:55 (1746074455) === conf-sanity: start cleanup 00:40:56 (1746074456) === === conf-sanity: finish cleanup 00:40:57 (1746074457) ===