-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 23:51:43 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg331-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg331-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg331-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg331-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg331-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:53:29 (1746071609) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg331-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg331-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg331-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1267 (time 1746071675.59 total 10.00 last 126.69) - open/close 2672 (time 1746071685.59 total 20.00 last 140.46) - open/close 3890 (time 1746071695.59 total 30.01 last 121.77) - open/close 5174 (time 1746071705.60 total 40.01 last 128.35) - open/close 6365 (time 1746071715.60 total 50.01 last 119.05) - open/close 7558 (time 1746071725.60 total 60.02 last 119.26) - open/close 8630 (time 1746071735.61 total 70.02 last 107.14) - open/close 9798 (time 1746071745.61 total 80.02 last 116.79) - open/close 11030 (time 1746071755.62 total 90.03 last 123.10) - open/close 12195 (time 1746071765.62 total 100.03 last 116.50) - open/close 13525 (time 1746071775.62 total 110.04 last 132.95) - open/close 14846 (time 1746071785.62 total 120.04 last 132.08) - open/close 16246 (time 1746071795.63 total 130.04 last 139.94) - open/close 17689 (time 1746071805.63 total 140.05 last 144.22) - open/close 19006 (time 1746071815.64 total 150.05 last 131.66) - open/close 20000 (time 1746071822.59 total 157.00 last 143.06) - open/close 21354 (time 1746071832.59 total 167.00 last 135.36) - open/close 22937 (time 1746071842.59 total 177.00 last 158.30) - open/close 24246 (time 1746071852.59 total 187.00 last 130.86) - open/close 25572 (time 1746071862.59 total 197.00 last 132.60) - open/close 27081 (time 1746071872.60 total 207.01 last 150.85) - open/close 28535 (time 1746071882.60 total 217.01 last 145.36) - open/close 30000 (time 1746071891.65 total 226.06 last 161.93) - open/close 31550 (time 1746071901.65 total 236.06 last 154.98) - open/close 32907 (time 1746071911.65 total 246.06 last 135.68) - open/close 34325 (time 1746071921.65 total 256.07 last 141.73) - open/close 35827 (time 1746071931.65 total 266.07 last 150.17) - open/close 37358 (time 1746071941.66 total 276.07 last 153.08) - open/close 38857 (time 1746071951.67 total 286.08 last 149.75) - open/close 40000 (time 1746071959.43 total 293.84 last 147.22) - open/close 41421 (time 1746071969.44 total 303.85 last 141.93) - open/close 42910 (time 1746071979.46 total 313.88 last 148.57) - open/close 44326 (time 1746071989.48 total 323.89 last 141.34) - open/close 45589 (time 1746071999.51 total 333.92 last 125.94) - open/close 46866 (time 1746072009.52 total 343.93 last 127.61) - open/close 48313 (time 1746072019.54 total 353.95 last 144.34) - open/close 49689 (time 1746072029.55 total 363.96 last 137.49) - open/close 50000 (time 1746072031.94 total 366.35 last 130.07) - open/close 51470 (time 1746072041.94 total 376.36 last 146.97) - open/close 52717 (time 1746072051.95 total 386.36 last 124.65) - open/close 53927 (time 1746072061.95 total 396.36 last 120.95) - open/close 55127 (time 1746072071.95 total 406.37 last 119.98) - open/close 56380 (time 1746072081.97 total 416.38 last 125.13) - open/close 57705 (time 1746072091.97 total 426.39 last 132.42) - open/close 59035 (time 1746072101.98 total 436.39 last 132.95) - open/close 60000 (time 1746072110.02 total 444.43 last 120.01) - open/close 61166 (time 1746072120.04 total 454.45 last 116.39) - open/close 62394 (time 1746072130.04 total 464.45 last 122.74) - open/close 63560 (time 1746072140.04 total 474.46 last 116.56) - open/close 64745 (time 1746072150.05 total 484.46 last 118.44) total: 64768 open/close in 484.68 seconds: 133.63 ops/second - /unlink 2115 (time 1746072162.27 total 10.01 last 211.38) - /unlink 3989 (time 1746072172.29 total 20.02 last 187.06) - /unlink 5850 (time 1746072182.29 total 30.02 last 186.10) - /unlink 7749 (time 1746072192.29 total 40.03 last 189.87) - /unlink 9939 (time 1746072202.29 total 50.03 last 218.96) - /unlink 11915 (time 1746072212.30 total 60.03 last 197.54) - /unlink 14151 (time 1746072222.30 total 70.03 last 223.55) - /unlink 16718 (time 1746072232.30 total 80.03 last 256.67) - /unlink 19231 (time 1746072242.30 total 90.03 last 251.30) - /unlink 20000 (time 1746072245.21 total 92.94 last 264.77) - /unlink 22398 (time 1746072255.21 total 102.94 last 239.80) - /unlink 24457 (time 1746072265.21 total 112.94 last 205.79) - /unlink 26802 (time 1746072275.21 total 122.95 last 234.48) - /unlink 29571 (time 1746072285.21 total 132.95 last 276.84) - /unlink 32210 (time 1746072295.22 total 142.95 last 263.82) - /unlink 34621 (time 1746072305.22 total 152.95 last 241.06) - /unlink 36708 (time 1746072315.25 total 162.98 last 208.03) - /unlink 38958 (time 1746072325.25 total 172.99 last 224.96) - /unlink 40000 (time 1746072328.75 total 176.48 last 298.06) - /unlink 42597 (time 1746072338.75 total 186.48 last 259.68) - /unlink 45072 (time 1746072348.76 total 196.49 last 247.32) - /unlink 47782 (time 1746072358.76 total 206.49 last 270.96) - /unlink 50000 (time 1746072365.84 total 213.58 last 313.04) - /unlink 53383 (time 1746072375.85 total 223.58 last 338.07) - /unlink 55842 (time 1746072385.86 total 233.59 last 245.63) - /unlink 58186 (time 1746072395.86 total 243.60 last 234.36) - /unlink 60000 (time 1746072403.81 total 251.54 last 228.39) - /unlink 62791 (time 1746072413.81 total 261.54 last 279.08) total: 64768 /unlink in 267.78 seconds: 241.87 ops/second Waiting for MDT destroys to complete - open/close 1408 (time 1746072436.78 total 10.01 last 140.70) - open/close 3164 (time 1746072446.79 total 20.01 last 175.53) - open/close 4681 (time 1746072456.79 total 30.02 last 151.64) - open/close 6375 (time 1746072466.79 total 40.02 last 169.36) - open/close 8031 (time 1746072476.80 total 50.02 last 165.59) - open/close 9661 (time 1746072486.80 total 60.02 last 162.96) - open/close 11245 (time 1746072496.80 total 70.02 last 158.35) - open/close 12915 (time 1746072506.81 total 80.03 last 166.92) - open/close 14226 (time 1746072516.82 total 90.04 last 130.93) - open/close 15609 (time 1746072526.82 total 100.04 last 138.26) - open/close 17032 (time 1746072536.83 total 110.05 last 142.24) - open/close 18338 (time 1746072546.83 total 120.05 last 130.52) - open/close 19714 (time 1746072556.84 total 130.06 last 137.49) - open/close 20000 (time 1746072559.05 total 132.28 last 129.10) - open/close 21580 (time 1746072569.06 total 142.28 last 157.93) - open/close 23198 (time 1746072579.07 total 152.29 last 161.70) - open/close 24720 (time 1746072589.07 total 162.29 last 152.15) - open/close 26268 (time 1746072599.07 total 172.30 last 154.72) - open/close 27777 (time 1746072609.08 total 182.30 last 150.87) - open/close 29224 (time 1746072619.08 total 192.31 last 144.58) - open/close 30000 (time 1746072623.62 total 196.84 last 171.04) - open/close 31554 (time 1746072633.63 total 206.85 last 155.33) - open/close 33157 (time 1746072643.63 total 216.85 last 160.28) - open/close 35173 (time 1746072653.63 total 226.85 last 201.51) - open/close 36660 (time 1746072663.64 total 236.86 last 148.64) - open/close 38277 (time 1746072673.64 total 246.86 last 161.63) - open/close 39603 (time 1746072683.65 total 256.87 last 132.51) - open/close 40000 (time 1746072686.73 total 259.95 last 128.83) - open/close 41347 (time 1746072696.73 total 269.95 last 134.67) - open/close 42763 (time 1746072706.73 total 279.96 last 141.56) - open/close 44140 (time 1746072716.74 total 289.96 last 137.66) - open/close 45532 (time 1746072726.74 total 299.96 last 139.16) - open/close 46866 (time 1746072736.74 total 309.96 last 133.40) - open/close 48238 (time 1746072746.74 total 319.96 last 137.19) - open/close 49575 (time 1746072756.74 total 329.96 last 133.69) - open/close 50000 (time 1746072760.22 total 333.45 last 122.05) - open/close 51330 (time 1746072770.22 total 343.45 last 132.98) - open/close 52860 (time 1746072780.23 total 353.45 last 152.94) - open/close 54165 (time 1746072790.23 total 363.46 last 130.45) - open/close 55409 (time 1746072800.24 total 373.46 last 124.34) - open/close 56754 (time 1746072810.24 total 383.46 last 134.45) - open/close 57995 (time 1746072820.24 total 393.47 last 124.07) - open/close 59471 (time 1746072830.25 total 403.47 last 147.49) - open/close 60000 (time 1746072833.93 total 407.15 last 143.72) - open/close 61250 (time 1746072843.93 total 417.16 last 124.96) - open/close 62632 (time 1746072853.94 total 427.16 last 138.14) - open/close 63931 (time 1746072863.94 total 437.17 last 129.86) total: 64768 open/close in 443.54 seconds: 146.03 ops/second - /unlink 1891 (time 1746072882.26 total 10.00 last 189.04) - /unlink 3963 (time 1746072892.26 total 20.01 last 207.16) - /unlink 6341 (time 1746072902.27 total 30.01 last 237.62) - /unlink 8621 (time 1746072912.28 total 40.02 last 227.92) - /unlink 10000 (time 1746072918.81 total 46.55 last 210.91) - /unlink 12088 (time 1746072928.83 total 56.57 last 208.56) - /unlink 14326 (time 1746072938.83 total 66.57 last 223.73) - /unlink 16436 (time 1746072948.83 total 76.57 last 210.94) - /unlink 18485 (time 1746072958.83 total 86.57 last 204.87) - /unlink 20000 (time 1746072965.20 total 92.94 last 238.12) - /unlink 22470 (time 1746072975.20 total 102.94 last 246.97) - /unlink 24658 (time 1746072985.20 total 112.94 last 218.73) - /unlink 26854 (time 1746072995.20 total 122.94 last 219.58) - /unlink 29124 (time 1746073005.20 total 132.94 last 226.95) - /unlink 30000 (time 1746073009.21 total 136.95 last 218.44) - /unlink 33118 (time 1746073019.21 total 146.96 last 311.76) - /unlink 35617 (time 1746073029.22 total 156.96 last 249.83) - /unlink 38050 (time 1746073039.24 total 166.98 last 242.80) - /unlink 40000 (time 1746073046.82 total 174.56 last 257.36) - /unlink 42449 (time 1746073056.82 total 184.56 last 244.85) - /unlink 45267 (time 1746073066.82 total 194.56 last 281.78) - /unlink 47866 (time 1746073076.83 total 204.57 last 259.72) - /unlink 50000 (time 1746073084.80 total 212.54 last 267.61) - /unlink 53260 (time 1746073094.80 total 222.54 last 325.99) - /unlink 56660 (time 1746073104.80 total 232.54 last 339.91) - /unlink 59605 (time 1746073114.80 total 242.55 last 294.43) - /unlink 62311 (time 1746073124.81 total 252.55 last 270.56) total: 64768 /unlink in 258.48 seconds: 250.57 ops/second Waiting for MDT destroys to complete - open/close 1899 (time 1746073152.97 total 10.00 last 189.88) - open/close 3405 (time 1746073162.97 total 20.01 last 150.54) - open/close 5176 (time 1746073172.98 total 30.01 last 177.03) - open/close 6782 (time 1746073182.98 total 40.01 last 160.59) - open/close 8239 (time 1746073192.98 total 50.01 last 145.65) - open/close 9896 (time 1746073202.98 total 60.01 last 165.69) - open/close 11235 (time 1746073212.99 total 70.02 last 133.82) - open/close 12687 (time 1746073223.00 total 80.03 last 145.03) - open/close 14255 (time 1746073233.00 total 90.03 last 156.76) - open/close 15900 (time 1746073243.01 total 100.05 last 164.26) - open/close 17434 (time 1746073253.02 total 110.05 last 153.37) - open/close 19160 (time 1746073263.02 total 120.05 last 172.55) - open/close 20000 (time 1746073268.37 total 125.40 last 156.99) - open/close 21556 (time 1746073278.37 total 135.41 last 155.56) - open/close 23198 (time 1746073288.38 total 145.41 last 164.13) - open/close 24591 (time 1746073298.38 total 155.42 last 139.24) - open/close 26171 (time 1746073308.38 total 165.42 last 157.96) - open/close 27576 (time 1746073318.39 total 175.43 last 140.39) - open/close 28880 (time 1746073328.40 total 185.43 last 130.30) - open/close 30000 (time 1746073336.71 total 193.74 last 134.84) - open/close 31269 (time 1746073346.71 total 203.74 last 126.85) - open/close 32605 (time 1746073356.72 total 213.75 last 133.51) - open/close 33976 (time 1746073366.72 total 223.75 last 137.06) - open/close 35208 (time 1746073376.73 total 233.76 last 123.10) - open/close 36553 (time 1746073386.73 total 243.76 last 134.49) - open/close 37861 (time 1746073396.73 total 253.77 last 130.75) - open/close 39172 (time 1746073406.74 total 263.77 last 131.05) - open/close 40000 (time 1746073414.71 total 271.75 last 103.82) - open/close 41414 (time 1746073424.72 total 281.75 last 141.32) - open/close 42843 (time 1746073434.72 total 291.75 last 142.85) - open/close 44346 (time 1746073444.72 total 301.76 last 150.29) - open/close 45720 (time 1746073454.73 total 311.76 last 137.35) - open/close 47067 (time 1746073464.73 total 321.76 last 134.64) - open/close 48439 (time 1746073474.73 total 331.76 last 137.20) - open/close 49848 (time 1746073484.74 total 341.77 last 140.83) - open/close 51329 (time 1746073494.74 total 351.77 last 148.10) - open/close 52910 (time 1746073504.74 total 361.77 last 158.03) - open/close 54217 (time 1746073514.74 total 371.77 last 130.70) - open/close 55616 (time 1746073524.74 total 381.77 last 139.89) - open/close 56996 (time 1746073534.75 total 391.79 last 137.83) - open/close 58482 (time 1746073544.76 total 401.80 last 148.44) - open/close 59937 (time 1746073554.76 total 411.80 last 145.49) - open/close 61254 (time 1746073564.77 total 421.80 last 131.64) - open/close 62413 (time 1746073574.77 total 431.81 last 115.85) - open/close 63510 (time 1746073584.77 total 441.81 last 109.70) - open/close 64543 (time 1746073594.78 total 451.81 last 103.28) total: 64768 open/close in 453.78 seconds: 142.73 ops/second - /unlink 2107 (time 1746073608.46 total 10.00 last 210.68) - /unlink 4070 (time 1746073618.46 total 20.00 last 196.27) - /unlink 5966 (time 1746073628.46 total 30.01 last 189.51) - /unlink 8099 (time 1746073638.47 total 40.01 last 213.22) - /unlink 9929 (time 1746073648.47 total 50.02 last 182.92) - /unlink 11942 (time 1746073658.48 total 60.02 last 201.20) - /unlink 14071 (time 1746073668.48 total 70.02 last 212.88) - /unlink 16088 (time 1746073678.48 total 80.03 last 201.62) - /unlink 17832 (time 1746073688.49 total 90.03 last 174.32) - /unlink 19735 (time 1746073698.49 total 100.03 last 190.29) - /unlink 21557 (time 1746073708.49 total 110.04 last 182.08) - /unlink 23472 (time 1746073718.49 total 120.04 last 191.50) - /unlink 25388 (time 1746073728.50 total 130.04 last 191.55) - /unlink 27282 (time 1746073738.50 total 140.04 last 189.35) - /unlink 29098 (time 1746073748.50 total 150.05 last 181.51) - /unlink 30000 (time 1746073753.45 total 155.00 last 182.30) - /unlink 31857 (time 1746073763.46 total 165.00 last 185.61) - /unlink 33754 (time 1746073773.46 total 175.01 last 189.56) - /unlink 35806 (time 1746073783.47 total 185.01 last 205.16) - /unlink 37896 (time 1746073793.47 total 195.01 last 208.93) - /unlink 40000 (time 1746073802.77 total 204.31 last 226.28) - /unlink 42262 (time 1746073812.77 total 214.32 last 226.09) - /unlink 44257 (time 1746073822.77 total 224.32 last 199.48) - /unlink 46196 (time 1746073832.77 total 234.32 last 193.89) - /unlink 48568 (time 1746073842.78 total 244.32 last 237.17) - /unlink 50000 (time 1746073849.91 total 251.45 last 200.82) - /unlink 51864 (time 1746073859.91 total 261.45 last 186.40) - /unlink 54168 (time 1746073869.91 total 271.46 last 230.27) - /unlink 56221 (time 1746073879.91 total 281.46 last 205.25) - /unlink 58179 (time 1746073889.92 total 291.46 last 195.79) - /unlink 60000 (time 1746073897.45 total 299.00 last 241.52) - /unlink 62482 (time 1746073907.46 total 309.00 last 248.13) - /unlink 64633 (time 1746073917.46 total 319.00 last 215.05) total: 64768 /unlink in 319.69 seconds: 202.60 ops/second Waiting for MDT destroys to complete - open/close 1246 (time 1746073934.51 total 10.00 last 124.56) - open/close 2497 (time 1746073944.51 total 20.01 last 125.08) - open/close 3485 (time 1746073954.51 total 30.01 last 98.80) - open/close 4704 (time 1746073964.52 total 40.01 last 121.80) - open/close 6208 (time 1746073974.52 total 50.01 last 150.39) - open/close 7550 (time 1746073984.52 total 60.02 last 134.17) - open/close 8904 (time 1746073994.53 total 70.02 last 135.29) - open/close 10000 (time 1746074004.46 total 79.96 last 110.31) - open/close 11172 (time 1746074014.47 total 89.96 last 117.15) - open/close 12299 (time 1746074024.47 total 99.97 last 112.65) - open/close 13602 (time 1746074034.47 total 109.97 last 130.29) - open/close 14817 (time 1746074044.48 total 119.98 last 121.42) - open/close 15910 (time 1746074054.48 total 129.98 last 109.26) - open/close 17168 (time 1746074064.48 total 139.98 last 125.80) - open/close 18272 (time 1746074074.49 total 149.98 last 110.34) - open/close 19345 (time 1746074084.50 total 160.00 last 107.12) - open/close 20000 (time 1746074090.76 total 166.25 last 104.76) - open/close 21129 (time 1746074100.76 total 176.25 last 112.90) - open/close 22343 (time 1746074110.76 total 186.26 last 121.35) - open/close 23583 (time 1746074120.77 total 196.26 last 123.92) - open/close 24706 (time 1746074130.77 total 206.27 last 112.28) - open/close 25872 (time 1746074140.77 total 216.27 last 116.58) - open/close 27140 (time 1746074150.77 total 226.27 last 126.75) - open/close 28107 (time 1746074160.78 total 236.28 last 96.65) - open/close 29198 (time 1746074170.78 total 246.28 last 109.03) - open/close 30000 (time 1746074177.85 total 253.35 last 113.44) - open/close 31164 (time 1746074187.86 total 263.35 last 116.39) - open/close 32267 (time 1746074197.86 total 273.36 last 110.29) - open/close 33314 (time 1746074207.86 total 283.36 last 104.65) - open/close 34429 (time 1746074217.87 total 293.36 last 111.47) - open/close 35761 (time 1746074227.87 total 303.37 last 133.13) - open/close 36844 (time 1746074237.87 total 313.37 last 108.28) - open/close 37986 (time 1746074247.88 total 323.38 last 114.11) - open/close 39211 (time 1746074257.88 total 333.38 last 122.47) - open/close 40000 (time 1746074264.86 total 340.36 last 112.99) - open/close 41043 (time 1746074274.88 total 350.38 last 104.17) - open/close 42213 (time 1746074284.88 total 360.38 last 117.00) - open/close 43476 (time 1746074294.88 total 370.38 last 126.28) - open/close 44538 (time 1746074304.88 total 380.38 last 106.18) - open/close 45688 (time 1746074314.89 total 390.38 last 114.95) - open/close 46961 (time 1746074324.89 total 400.39 last 127.25) - open/close 48003 (time 1746074334.90 total 410.40 last 104.11) - open/close 49048 (time 1746074344.90 total 420.40 last 104.44) - open/close 50000 (time 1746074352.60 total 428.10 last 123.69) - open/close 51203 (time 1746074362.60 total 438.10 last 120.30) - open/close 52482 (time 1746074372.60 total 448.10 last 127.89) - open/close 53816 (time 1746074382.60 total 458.10 last 133.39) - open/close 55241 (time 1746074392.60 total 468.10 last 142.48) - open/close 56498 (time 1746074402.60 total 478.10 last 125.69) - open/close 57794 (time 1746074412.61 total 488.11 last 129.49) - open/close 59103 (time 1746074422.63 total 498.13 last 130.62) - open/close 60000 (time 1746074430.52 total 506.02 last 113.75) - open/close 61098 (time 1746074440.53 total 516.03 last 109.69) - open/close 62325 (time 1746074450.53 total 526.03 last 122.65) - open/close 63644 (time 1746074460.55 total 536.05 last 131.71) total: 64768 open/close in 545.31 seconds: 118.77 ops/second - /unlink 2140 (time 1746074481.55 total 10.00 last 213.92) - /unlink 4422 (time 1746074491.55 total 20.00 last 228.19) - /unlink 6550 (time 1746074501.56 total 30.01 last 212.74) - /unlink 8869 (time 1746074511.56 total 40.01 last 231.84) - /unlink 10000 (time 1746074516.53 total 44.98 last 227.36) - /unlink 12224 (time 1746074526.54 total 54.98 last 222.37) - /unlink 14589 (time 1746074536.54 total 64.99 last 236.43) - /unlink 16931 (time 1746074546.54 total 74.99 last 234.15) - /unlink 19040 (time 1746074556.54 total 84.99 last 210.81) - /unlink 20000 (time 1746074560.98 total 89.43 last 216.55) - /unlink 22387 (time 1746074570.98 total 99.43 last 238.69) - /unlink 24583 (time 1746074580.98 total 109.43 last 219.47) - /unlink 26682 (time 1746074590.99 total 119.44 last 209.83) - /unlink 29001 (time 1746074600.99 total 129.44 last 231.81) - /unlink 30000 (time 1746074605.65 total 134.10 last 214.54) - /unlink 32373 (time 1746074615.65 total 144.10 last 237.20) - /unlink 34797 (time 1746074625.65 total 154.10 last 242.38) - /unlink 36809 (time 1746074635.65 total 164.10 last 201.15) - /unlink 38953 (time 1746074645.70 total 174.15 last 213.35) - /unlink 40000 (time 1746074650.51 total 178.96 last 218.04) - /unlink 42400 (time 1746074660.52 total 188.97 last 239.61) - /unlink 44516 (time 1746074670.52 total 198.97 last 211.59) - /unlink 46858 (time 1746074680.52 total 208.97 last 234.15) - /unlink 49606 (time 1746074690.53 total 218.98 last 274.76) - /unlink 52048 (time 1746074700.53 total 228.98 last 244.16) - /unlink 54388 (time 1746074710.53 total 238.98 last 233.97) - /unlink 56511 (time 1746074720.53 total 248.98 last 212.29) - /unlink 59002 (time 1746074730.53 total 258.98 last 249.04) - /unlink 60000 (time 1746074734.59 total 263.04 last 245.63) - /unlink 62347 (time 1746074744.60 total 273.05 last 234.65) - /unlink 64755 (time 1746074754.60 total 283.05 last 240.73) total: 64768 /unlink in 283.11 seconds: 228.77 ops/second Waiting for MDT destroys to complete - open/close 1492 (time 1746074774.94 total 10.00 last 149.15) - open/close 2999 (time 1746074784.94 total 20.01 last 150.66) - open/close 4422 (time 1746074794.95 total 30.01 last 142.21) - open/close 5779 (time 1746074804.95 total 40.01 last 135.68) - open/close 7252 (time 1746074814.96 total 50.02 last 147.23) - open/close 8716 (time 1746074824.96 total 60.02 last 146.36) - open/close 10000 (time 1746074833.35 total 68.42 last 152.96) - open/close 11531 (time 1746074843.36 total 78.42 last 153.05) - open/close 13027 (time 1746074853.36 total 88.42 last 149.52) - open/close 14820 (time 1746074863.36 total 98.43 last 179.23) - open/close 16422 (time 1746074873.37 total 108.44 last 160.05) - open/close 17992 (time 1746074883.38 total 118.44 last 156.89) - open/close 19468 (time 1746074893.38 total 128.45 last 147.57) - open/close 20000 (time 1746074896.96 total 132.02 last 148.92) - open/close 21521 (time 1746074906.97 total 142.03 last 151.91) - open/close 22765 (time 1746074916.97 total 152.03 last 124.36) - open/close 24061 (time 1746074926.98 total 162.04 last 129.52) - open/close 25401 (time 1746074936.98 total 172.04 last 133.97) - open/close 26659 (time 1746074946.98 total 182.04 last 125.77) - open/close 27871 (time 1746074956.99 total 192.05 last 121.15) - open/close 29127 (time 1746074967.00 total 202.06 last 125.40) - open/close 30000 (time 1746074973.57 total 208.64 last 132.86) - open/close 31353 (time 1746074983.58 total 218.64 last 135.21) - open/close 32635 (time 1746074993.59 total 228.65 last 128.09) - open/close 34019 (time 1746075003.59 total 238.65 last 138.39) - open/close 35285 (time 1746075013.59 total 248.65 last 126.56) - open/close 36592 (time 1746075023.59 total 258.66 last 130.69) - open/close 37865 (time 1746075033.60 total 268.67 last 127.16) - open/close 39187 (time 1746075043.61 total 278.67 last 132.16) - open/close 40000 (time 1746075050.09 total 285.15 last 125.44) - open/close 41376 (time 1746075060.09 total 295.15 last 137.60) - open/close 42718 (time 1746075070.09 total 305.15 last 134.18) - open/close 43976 (time 1746075080.09 total 315.16 last 125.77) - open/close 45290 (time 1746075090.09 total 325.16 last 131.36) - open/close 46563 (time 1746075100.10 total 335.16 last 127.30) - open/close 47791 (time 1746075110.10 total 345.16 last 122.75) - open/close 49036 (time 1746075120.11 total 355.17 last 124.42) - open/close 50000 (time 1746075127.29 total 362.35 last 134.16) - open/close 51371 (time 1746075137.30 total 372.36 last 137.02) - open/close 52602 (time 1746075147.31 total 382.37 last 122.95) - open/close 53895 (time 1746075157.31 total 392.38 last 129.26) - open/close 55213 (time 1746075167.32 total 402.39 last 131.67) - open/close 56432 (time 1746075177.32 total 412.39 last 121.88) - open/close 57731 (time 1746075187.33 total 422.39 last 129.84) - open/close 59030 (time 1746075197.33 total 432.39 last 129.86) - open/close 60000 (time 1746075205.77 total 440.83 last 114.94) - open/close 61265 (time 1746075215.78 total 450.84 last 126.42) - open/close 62545 (time 1746075225.78 total 460.84 last 127.97) - open/close 63835 (time 1746075235.79 total 470.86 last 128.83) total: 64768 open/close in 479.03 seconds: 135.21 ops/second - /unlink 1994 (time 1746075255.92 total 10.01 last 199.15) - /unlink 4072 (time 1746075265.92 total 20.02 last 207.68) - /unlink 6017 (time 1746075275.92 total 30.02 last 194.45) - /unlink 7798 (time 1746075285.93 total 40.03 last 177.92) - /unlink 9998 (time 1746075295.94 total 50.03 last 219.97) - /unlink 11972 (time 1746075305.94 total 60.03 last 197.38) - /unlink 13790 (time 1746075315.94 total 70.04 last 181.73) - /unlink 15686 (time 1746075325.94 total 80.04 last 189.58) - /unlink 17821 (time 1746075335.95 total 90.05 last 213.28) - /unlink 19657 (time 1746075345.96 total 100.05 last 183.53) - /unlink 21748 (time 1746075355.96 total 110.05 last 209.05) - /unlink 23839 (time 1746075365.96 total 120.06 last 209.02) - /unlink 25920 (time 1746075375.97 total 130.06 last 208.01) - /unlink 27917 (time 1746075385.97 total 140.06 last 199.67) - /unlink 30000 (time 1746075395.13 total 149.23 last 227.23) - /unlink 31942 (time 1746075405.14 total 159.23 last 194.14) - /unlink 34007 (time 1746075415.15 total 169.24 last 206.31) - /unlink 35778 (time 1746075425.15 total 179.25 last 177.06) - /unlink 37560 (time 1746075435.15 total 189.25 last 178.12) - /unlink 39761 (time 1746075445.16 total 199.25 last 220.03) - /unlink 42043 (time 1746075455.16 total 209.26 last 228.14) - /unlink 44445 (time 1746075465.16 total 219.26 last 240.10) - /unlink 46827 (time 1746075475.16 total 229.26 last 238.18) - /unlink 49019 (time 1746075485.17 total 239.27 last 219.02) - /unlink 50000 (time 1746075489.69 total 243.79 last 217.14) - /unlink 52159 (time 1746075499.69 total 253.79 last 215.86) - /unlink 54364 (time 1746075509.69 total 263.79 last 220.46) - /unlink 56405 (time 1746075519.70 total 273.79 last 204.03) - /unlink 58344 (time 1746075529.70 total 283.80 last 193.87) - /unlink 60000 (time 1746075537.90 total 291.99 last 202.02) - /unlink 62152 (time 1746075547.90 total 302.00 last 215.14) - /unlink 64351 (time 1746075557.91 total 312.00 last 219.76) total: 64768 /unlink in 313.99 seconds: 206.27 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 106 (3991s) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4134 sec ========== 01:00:38 (1746075638) === conf-sanity: start cleanup 01:00:39 (1746075639) === === conf-sanity: finish cleanup 01:00:41 (1746075641) ===