-----============= acceptance-small: conf-sanity ============----- Thu May 1 22:53:30 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg427-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg427-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg427-server oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_size: failed to read coresize from /sys/module/acpi_cpufreq libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg427-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg427-server: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg427-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg427-server: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg427-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:55:15 (1746154515) Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg427-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1212 (time 1746154590.75 total 10.00 last 121.18) - open/close 2562 (time 1746154600.76 total 20.01 last 134.92) - open/close 3705 (time 1746154610.76 total 30.01 last 114.25) - open/close 4882 (time 1746154620.77 total 40.02 last 117.65) - open/close 6140 (time 1746154630.77 total 50.02 last 125.78) - open/close 7475 (time 1746154640.78 total 60.02 last 133.41) - open/close 8888 (time 1746154650.78 total 70.03 last 141.27) - open/close 10000 (time 1746154658.41 total 77.66 last 145.76) - open/close 11478 (time 1746154668.41 total 87.66 last 147.72) - open/close 12811 (time 1746154678.42 total 97.66 last 133.27) - open/close 14382 (time 1746154688.42 total 107.66 last 157.08) - open/close 15671 (time 1746154698.42 total 117.67 last 128.86) - open/close 17129 (time 1746154708.42 total 127.67 last 145.75) - open/close 18343 (time 1746154718.42 total 137.67 last 121.38) - open/close 19800 (time 1746154728.43 total 147.68 last 145.59) - open/close 20986 (time 1746154738.44 total 157.69 last 118.53) - open/close 22113 (time 1746154748.44 total 167.69 last 112.69) - open/close 23249 (time 1746154758.44 total 177.69 last 113.59) - open/close 24446 (time 1746154768.45 total 187.70 last 119.54) - open/close 25633 (time 1746154778.46 total 197.71 last 118.62) - open/close 26912 (time 1746154788.46 total 207.71 last 127.89) - open/close 28210 (time 1746154798.46 total 217.71 last 129.76) - open/close 29497 (time 1746154808.47 total 227.72 last 128.64) - open/close 30000 (time 1746154812.27 total 231.52 last 132.11) - open/close 31358 (time 1746154822.28 total 241.53 last 135.77) - open/close 32720 (time 1746154832.28 total 251.53 last 136.18) - open/close 33842 (time 1746154842.29 total 261.54 last 112.11) - open/close 35244 (time 1746154852.29 total 271.54 last 140.14) - open/close 36527 (time 1746154862.31 total 281.55 last 128.11) - open/close 37781 (time 1746154872.32 total 291.57 last 125.25) - open/close 39023 (time 1746154882.32 total 301.57 last 124.18) - open/close 40000 (time 1746154891.53 total 310.78 last 106.10) - open/close 41150 (time 1746154901.53 total 320.78 last 115.00) - open/close 42401 (time 1746154911.54 total 330.79 last 124.97) - open/close 43427 (time 1746154921.54 total 340.79 last 102.58) - open/close 44657 (time 1746154931.55 total 350.80 last 122.88) - open/close 45943 (time 1746154941.55 total 360.80 last 128.54) - open/close 47156 (time 1746154951.56 total 370.81 last 121.23) - open/close 48295 (time 1746154961.56 total 380.81 last 113.90) - open/close 49384 (time 1746154971.56 total 390.81 last 108.89) - open/close 50000 (time 1746154977.13 total 396.38 last 110.59) - open/close 51193 (time 1746154987.13 total 406.38 last 119.26) - open/close 52666 (time 1746154997.13 total 416.38 last 147.30) - open/close 53987 (time 1746155007.14 total 426.39 last 132.01) - open/close 55517 (time 1746155017.15 total 436.40 last 152.93) - open/close 56640 (time 1746155027.15 total 446.40 last 112.27) - open/close 57882 (time 1746155037.16 total 456.40 last 124.12) - open/close 59285 (time 1746155047.16 total 466.41 last 140.25) - open/close 60000 (time 1746155052.98 total 472.23 last 122.82) - open/close 61239 (time 1746155062.98 total 482.23 last 123.89) - open/close 62574 (time 1746155072.98 total 492.23 last 133.46) - open/close 64076 (time 1746155082.98 total 502.23 last 150.19) total: 64768 open/close in 507.71 seconds: 127.57 ops/second - /unlink 2936 (time 1746155100.03 total 10.00 last 293.48) - /unlink 5570 (time 1746155110.03 total 20.00 last 263.38) - /unlink 7830 (time 1746155120.03 total 30.01 last 226.00) - /unlink 10000 (time 1746155128.95 total 38.92 last 243.35) - /unlink 12777 (time 1746155138.95 total 48.93 last 277.62) - /unlink 15180 (time 1746155148.96 total 58.93 last 240.15) - /unlink 17552 (time 1746155158.96 total 68.93 last 237.19) - /unlink 20000 (time 1746155168.85 total 78.82 last 247.56) - /unlink 22731 (time 1746155178.85 total 88.82 last 273.07) - /unlink 25121 (time 1746155188.85 total 98.82 last 238.93) - /unlink 27322 (time 1746155198.85 total 108.83 last 220.07) - /unlink 29769 (time 1746155208.86 total 118.83 last 244.57) - /unlink 31705 (time 1746155218.87 total 128.84 last 193.45) - /unlink 33985 (time 1746155228.87 total 138.84 last 227.95) - /unlink 36134 (time 1746155238.87 total 148.84 last 214.89) - /unlink 38361 (time 1746155248.88 total 158.85 last 222.53) - /unlink 40000 (time 1746155255.18 total 165.15 last 259.97) - /unlink 42656 (time 1746155265.18 total 175.16 last 265.57) - /unlink 45587 (time 1746155275.18 total 185.16 last 293.06) - /unlink 48124 (time 1746155285.19 total 195.16 last 253.60) - /unlink 50000 (time 1746155292.82 total 202.79 last 245.88) - /unlink 52363 (time 1746155302.82 total 212.79 last 236.26) - /unlink 54601 (time 1746155312.82 total 222.79 last 223.79) - /unlink 56741 (time 1746155322.82 total 232.79 last 213.95) - /unlink 59056 (time 1746155332.82 total 242.80 last 231.43) - /unlink 60000 (time 1746155336.83 total 246.81 last 235.50) - /unlink 62382 (time 1746155346.83 total 256.81 last 238.18) total: 64768 /unlink in 266.12 seconds: 243.38 ops/second Waiting for MDT destroys to complete - open/close 1283 (time 1746155371.67 total 10.00 last 128.28) - open/close 2612 (time 1746155381.67 total 20.01 last 132.84) - open/close 3944 (time 1746155391.68 total 30.01 last 133.12) - open/close 5312 (time 1746155401.68 total 40.01 last 136.78) - open/close 6635 (time 1746155411.68 total 50.02 last 132.22) - open/close 7860 (time 1746155421.68 total 60.02 last 122.49) - open/close 9146 (time 1746155431.69 total 70.02 last 128.55) - open/close 10000 (time 1746155438.22 total 76.56 last 130.76) - open/close 11467 (time 1746155448.22 total 86.56 last 146.63) - open/close 12677 (time 1746155458.24 total 96.57 last 120.83) - open/close 13859 (time 1746155468.24 total 106.58 last 118.18) - open/close 15156 (time 1746155478.25 total 116.58 last 129.61) - open/close 16303 (time 1746155488.26 total 126.60 last 114.55) - open/close 17359 (time 1746155498.26 total 136.60 last 105.59) - open/close 18593 (time 1746155508.26 total 146.60 last 123.40) - open/close 20000 (time 1746155516.97 total 155.31 last 161.53) - open/close 21393 (time 1746155526.97 total 165.31 last 139.30) - open/close 22687 (time 1746155536.98 total 175.31 last 129.34) - open/close 24017 (time 1746155546.98 total 185.32 last 132.96) - open/close 25367 (time 1746155556.98 total 195.32 last 134.97) - open/close 26635 (time 1746155566.98 total 205.32 last 126.78) - open/close 27940 (time 1746155576.98 total 215.32 last 130.49) - open/close 29267 (time 1746155586.99 total 225.32 last 132.67) - open/close 30000 (time 1746155592.79 total 231.12 last 126.33) - open/close 31636 (time 1746155602.79 total 241.13 last 163.58) - open/close 32976 (time 1746155612.80 total 251.13 last 133.90) - open/close 34244 (time 1746155622.80 total 261.14 last 126.77) - open/close 35505 (time 1746155632.82 total 271.15 last 125.88) - open/close 36816 (time 1746155642.83 total 281.17 last 130.92) - open/close 38021 (time 1746155652.84 total 291.17 last 120.43) - open/close 39238 (time 1746155662.84 total 301.18 last 121.61) - open/close 40000 (time 1746155668.27 total 306.60 last 140.52) - open/close 41684 (time 1746155678.27 total 316.60 last 168.39) - open/close 43139 (time 1746155688.27 total 326.60 last 145.48) - open/close 44814 (time 1746155698.27 total 336.61 last 167.46) - open/close 46429 (time 1746155708.28 total 346.62 last 161.30) - open/close 47831 (time 1746155718.29 total 356.62 last 140.16) - open/close 48940 (time 1746155728.29 total 366.62 last 110.87) - open/close 50000 (time 1746155737.87 total 376.21 last 110.64) - open/close 51122 (time 1746155747.87 total 386.21 last 112.17) - open/close 52252 (time 1746155757.87 total 396.21 last 112.96) - open/close 53409 (time 1746155767.88 total 406.21 last 115.69) - open/close 54579 (time 1746155777.88 total 416.22 last 116.94) - open/close 55775 (time 1746155787.88 total 426.22 last 119.55) - open/close 56924 (time 1746155797.89 total 436.23 last 114.83) - open/close 57990 (time 1746155807.90 total 446.23 last 106.55) - open/close 59062 (time 1746155817.90 total 456.23 last 107.18) - open/close 60000 (time 1746155825.88 total 464.22 last 117.49) - open/close 61121 (time 1746155835.89 total 474.22 last 112.04) - open/close 62258 (time 1746155845.89 total 484.23 last 113.64) - open/close 63477 (time 1746155855.90 total 494.23 last 121.84) - open/close 64590 (time 1746155865.91 total 504.24 last 111.17) total: 64768 open/close in 506.31 seconds: 127.92 ops/second - /unlink 1469 (time 1746155879.99 total 10.00 last 146.89) - /unlink 3490 (time 1746155889.99 total 20.00 last 202.04) - /unlink 5114 (time 1746155899.99 total 30.01 last 162.38) - /unlink 7016 (time 1746155909.99 total 40.01 last 190.18) - /unlink 9328 (time 1746155920.00 total 50.01 last 231.07) - /unlink 10000 (time 1746155923.95 total 53.96 last 170.01) - /unlink 11935 (time 1746155933.95 total 63.97 last 193.46) - /unlink 13889 (time 1746155943.95 total 73.97 last 195.32) - /unlink 15713 (time 1746155953.96 total 83.97 last 182.35) - /unlink 17581 (time 1746155963.96 total 93.98 last 186.75) - /unlink 19480 (time 1746155973.97 total 103.98 last 189.80) - /unlink 20000 (time 1746155976.76 total 106.77 last 186.12) - /unlink 21725 (time 1746155986.76 total 116.78 last 172.43) - /unlink 23486 (time 1746155996.76 total 126.78 last 176.08) - /unlink 25182 (time 1746156006.77 total 136.79 last 169.50) - /unlink 26999 (time 1746156016.78 total 146.79 last 181.54) - /unlink 29011 (time 1746156026.79 total 156.81 last 200.97) - /unlink 30000 (time 1746156031.82 total 161.83 last 196.78) - /unlink 31857 (time 1746156041.82 total 171.83 last 185.68) - /unlink 33836 (time 1746156051.82 total 181.84 last 197.83) - /unlink 36026 (time 1746156061.82 total 191.84 last 218.96) - /unlink 37810 (time 1746156071.83 total 201.84 last 178.34) - /unlink 39626 (time 1746156081.84 total 211.86 last 181.35) - /unlink 41649 (time 1746156091.85 total 221.86 last 202.16) - /unlink 43779 (time 1746156101.85 total 231.86 last 212.96) - /unlink 46017 (time 1746156111.85 total 241.87 last 223.76) - /unlink 47857 (time 1746156121.85 total 251.87 last 183.99) - /unlink 49611 (time 1746156131.85 total 261.87 last 175.37) - /unlink 51801 (time 1746156141.86 total 271.87 last 218.95) - /unlink 53736 (time 1746156151.86 total 281.88 last 193.41) - /unlink 55658 (time 1746156161.86 total 291.88 last 192.18) - /unlink 57667 (time 1746156171.86 total 301.88 last 200.85) - /unlink 59626 (time 1746156181.86 total 311.88 last 195.88) - /unlink 60000 (time 1746156183.88 total 313.89 last 185.90) - /unlink 61812 (time 1746156193.88 total 323.90 last 181.07) - /unlink 63539 (time 1746156203.89 total 333.90 last 172.60) total: 64768 /unlink in 340.58 seconds: 190.17 ops/second Waiting for MDT destroys to complete - open/close 1102 (time 1746156232.56 total 10.00 last 110.18) - open/close 2346 (time 1746156242.57 total 20.01 last 124.29) - open/close 3393 (time 1746156252.58 total 30.02 last 104.63) - open/close 4490 (time 1746156262.59 total 40.02 last 109.64) - open/close 5703 (time 1746156272.59 total 50.03 last 121.27) - open/close 6803 (time 1746156282.60 total 60.03 last 109.92) - open/close 7978 (time 1746156292.60 total 70.04 last 117.42) - open/close 9149 (time 1746156302.61 total 80.05 last 117.01) - open/close 10000 (time 1746156310.48 total 87.92 last 108.06) - open/close 11159 (time 1746156320.49 total 97.93 last 115.84) - open/close 12339 (time 1746156330.49 total 107.93 last 117.98) - open/close 13356 (time 1746156340.49 total 117.93 last 101.69) - open/close 14404 (time 1746156350.50 total 127.94 last 104.73) - open/close 15536 (time 1746156360.50 total 137.94 last 113.19) - open/close 16667 (time 1746156370.54 total 147.97 last 112.69) - open/close 17811 (time 1746156380.54 total 157.98 last 114.38) - open/close 18881 (time 1746156390.54 total 167.98 last 106.94) - open/close 19929 (time 1746156400.54 total 177.98 last 104.79) - open/close 20925 (time 1746156410.55 total 187.99 last 99.56) - open/close 22066 (time 1746156420.55 total 197.99 last 114.08) - open/close 23240 (time 1746156430.55 total 207.99 last 117.38) - open/close 24362 (time 1746156440.56 total 217.99 last 112.15) - open/close 25623 (time 1746156450.56 total 228.00 last 126.03) - open/close 26907 (time 1746156460.57 total 238.00 last 128.32) - open/close 28021 (time 1746156470.57 total 248.01 last 111.39) - open/close 29274 (time 1746156480.58 total 258.01 last 125.18) - open/close 30000 (time 1746156486.24 total 263.68 last 128.20) - open/close 31092 (time 1746156496.25 total 273.69 last 109.06) - open/close 32291 (time 1746156506.25 total 283.69 last 119.90) - open/close 33519 (time 1746156516.26 total 293.69 last 122.75) - open/close 34778 (time 1746156526.26 total 303.70 last 125.89) - open/close 36060 (time 1746156536.26 total 313.70 last 128.19) - open/close 37299 (time 1746156546.26 total 323.70 last 123.90) - open/close 38764 (time 1746156556.26 total 333.70 last 146.49) - open/close 40000 (time 1746156564.77 total 342.21 last 145.19) - open/close 41265 (time 1746156574.78 total 352.22 last 126.37) - open/close 42606 (time 1746156584.79 total 362.23 last 133.96) - open/close 43865 (time 1746156594.80 total 372.24 last 125.83) - open/close 44996 (time 1746156604.80 total 382.24 last 113.03) - open/close 46381 (time 1746156614.81 total 392.25 last 138.41) - open/close 47693 (time 1746156624.82 total 402.26 last 131.08) - open/close 48905 (time 1746156634.83 total 412.26 last 121.12) - open/close 50000 (time 1746156643.52 total 420.96 last 125.92) - open/close 51165 (time 1746156653.53 total 430.96 last 116.46) - open/close 52342 (time 1746156663.53 total 440.97 last 117.66) - open/close 53573 (time 1746156673.54 total 450.97 last 123.00) - open/close 54847 (time 1746156683.54 total 460.98 last 127.33) - open/close 55989 (time 1746156693.54 total 470.98 last 114.17) - open/close 57203 (time 1746156703.55 total 480.99 last 121.32) - open/close 58389 (time 1746156713.56 total 491.00 last 118.50) - open/close 59468 (time 1746156723.57 total 501.01 last 107.81) - open/close 60000 (time 1746156728.42 total 505.85 last 109.75) - open/close 61065 (time 1746156738.43 total 515.87 last 106.35) - open/close 62160 (time 1746156748.44 total 525.88 last 109.36) - open/close 63170 (time 1746156758.44 total 535.88 last 100.98) - open/close 64254 (time 1746156768.45 total 545.88 last 108.38) total: 64768 open/close in 550.81 seconds: 117.59 ops/second - /unlink 2034 (time 1746156784.95 total 10.00 last 203.32) - /unlink 3731 (time 1746156794.95 total 20.00 last 169.69) - /unlink 5743 (time 1746156804.95 total 30.00 last 201.19) - /unlink 7950 (time 1746156814.95 total 40.01 last 220.68) - /unlink 10000 (time 1746156824.34 total 49.40 last 218.27) - /unlink 12532 (time 1746156834.35 total 59.40 last 253.11) - /unlink 15178 (time 1746156844.35 total 69.40 last 264.57) - /unlink 17606 (time 1746156854.35 total 79.41 last 242.70) - /unlink 20000 (time 1746156864.34 total 89.40 last 239.65) - /unlink 22452 (time 1746156874.35 total 99.40 last 245.13) - /unlink 24729 (time 1746156884.35 total 109.40 last 227.60) - /unlink 27031 (time 1746156894.35 total 119.40 last 230.20) - /unlink 29581 (time 1746156904.36 total 129.41 last 254.76) - /unlink 32542 (time 1746156914.36 total 139.41 last 296.08) - /unlink 35443 (time 1746156924.36 total 149.42 last 290.03) - /unlink 38364 (time 1746156934.36 total 159.42 last 292.04) - /unlink 40000 (time 1746156940.64 total 165.69 last 260.83) - /unlink 42680 (time 1746156950.64 total 175.69 last 267.96) - /unlink 45250 (time 1746156960.64 total 185.70 last 256.88) - /unlink 47712 (time 1746156970.65 total 195.70 last 246.12) - /unlink 50000 (time 1746156980.05 total 205.10 last 243.36) - /unlink 52733 (time 1746156990.05 total 215.10 last 273.29) - /unlink 55519 (time 1746157000.05 total 225.11 last 278.49) - /unlink 58054 (time 1746157010.05 total 235.11 last 253.46) - /unlink 60000 (time 1746157017.68 total 242.73 last 255.30) - /unlink 62740 (time 1746157027.68 total 252.73 last 273.91) total: 64768 /unlink in 260.70 seconds: 248.44 ops/second Waiting for MDT destroys to complete - open/close 1334 (time 1746157058.78 total 10.00 last 133.37) - open/close 2622 (time 1746157068.78 total 20.01 last 128.76) - open/close 4121 (time 1746157078.79 total 30.01 last 149.79) - open/close 5845 (time 1746157088.79 total 40.01 last 172.37) - open/close 7665 (time 1746157098.80 total 50.02 last 181.95) - open/close 9422 (time 1746157108.80 total 60.02 last 175.65) - open/close 10000 (time 1746157112.17 total 63.40 last 171.26) - open/close 11729 (time 1746157122.18 total 73.40 last 172.87) - open/close 13420 (time 1746157132.18 total 83.40 last 169.07) - open/close 15104 (time 1746157142.18 total 93.40 last 168.30) - open/close 16801 (time 1746157152.19 total 103.41 last 169.65) - open/close 18611 (time 1746157162.19 total 113.41 last 180.96) - open/close 20000 (time 1746157170.44 total 121.66 last 168.33) - open/close 21658 (time 1746157180.44 total 131.66 last 165.80) - open/close 23038 (time 1746157190.44 total 141.66 last 137.99) - open/close 24707 (time 1746157200.45 total 151.67 last 166.80) - open/close 26260 (time 1746157210.46 total 161.68 last 155.12) - open/close 27881 (time 1746157220.46 total 171.68 last 162.03) - open/close 29537 (time 1746157230.47 total 181.69 last 165.52) - open/close 30000 (time 1746157233.32 total 184.54 last 162.51) - open/close 31424 (time 1746157243.32 total 194.54 last 142.39) - open/close 33029 (time 1746157253.32 total 204.54 last 160.49) - open/close 34532 (time 1746157263.32 total 214.54 last 150.27) - open/close 36017 (time 1746157273.32 total 224.54 last 148.47) - open/close 37650 (time 1746157283.33 total 234.55 last 163.22) - open/close 39230 (time 1746157293.33 total 244.55 last 157.98) - open/close 40000 (time 1746157298.21 total 249.43 last 157.89) - open/close 41304 (time 1746157308.22 total 259.44 last 130.28) - open/close 42405 (time 1746157318.22 total 269.44 last 110.04) - open/close 43512 (time 1746157328.23 total 279.45 last 110.60) - open/close 44660 (time 1746157338.23 total 289.45 last 114.74) - open/close 45767 (time 1746157348.24 total 299.46 last 110.67) - open/close 46874 (time 1746157358.24 total 309.46 last 110.69) - open/close 48409 (time 1746157368.24 total 319.46 last 153.47) - open/close 50000 (time 1746157377.82 total 329.04 last 166.10) - open/close 51517 (time 1746157387.82 total 339.04 last 151.68) - open/close 53225 (time 1746157397.82 total 349.04 last 170.77) - open/close 54744 (time 1746157407.83 total 359.05 last 151.83) - open/close 56167 (time 1746157417.83 total 369.05 last 142.25) - open/close 57733 (time 1746157427.84 total 379.06 last 156.53) - open/close 59310 (time 1746157437.84 total 389.06 last 157.59) - open/close 60000 (time 1746157442.71 total 393.93 last 141.84) - open/close 61338 (time 1746157452.71 total 403.93 last 133.77) - open/close 62944 (time 1746157462.71 total 413.94 last 160.51) - open/close 64501 (time 1746157472.72 total 423.94 last 155.63) total: 64768 open/close in 425.76 seconds: 152.12 ops/second - /unlink 2422 (time 1746157486.20 total 10.00 last 242.12) - /unlink 4751 (time 1746157496.20 total 20.00 last 232.89) - /unlink 7003 (time 1746157506.20 total 30.01 last 225.15) - /unlink 9368 (time 1746157516.20 total 40.01 last 236.40) - /unlink 10000 (time 1746157519.22 total 43.02 last 209.81) - /unlink 12234 (time 1746157529.22 total 53.02 last 223.34) - /unlink 14471 (time 1746157539.22 total 63.03 last 223.60) - /unlink 17188 (time 1746157549.23 total 73.03 last 271.62) - /unlink 19647 (time 1746157559.23 total 83.03 last 245.88) - /unlink 21984 (time 1746157569.23 total 93.03 last 233.69) - /unlink 24217 (time 1746157579.23 total 103.04 last 223.25) - /unlink 26330 (time 1746157589.23 total 113.04 last 211.27) - /unlink 28803 (time 1746157599.23 total 123.04 last 247.25) - /unlink 30000 (time 1746157603.58 total 127.38 last 275.63) - /unlink 32697 (time 1746157613.58 total 137.38 last 269.68) - /unlink 34950 (time 1746157623.58 total 147.38 last 225.28) - /unlink 37227 (time 1746157633.58 total 157.38 last 227.70) - /unlink 39707 (time 1746157643.58 total 167.39 last 247.97) - /unlink 42057 (time 1746157653.58 total 177.39 last 234.99) - /unlink 44431 (time 1746157663.59 total 187.39 last 237.24) - /unlink 46872 (time 1746157673.59 total 197.40 last 244.03) - /unlink 49374 (time 1746157683.59 total 207.40 last 250.07) - /unlink 50000 (time 1746157686.37 total 210.17 last 225.88) - /unlink 52254 (time 1746157696.37 total 220.17 last 225.40) - /unlink 54731 (time 1746157706.37 total 230.17 last 247.69) - /unlink 57146 (time 1746157716.37 total 240.18 last 241.34) - /unlink 59575 (time 1746157726.37 total 250.18 last 242.87) - /unlink 61984 (time 1746157736.37 total 260.18 last 240.90) - /unlink 64368 (time 1746157746.38 total 270.18 last 238.38) total: 64768 /unlink in 271.54 seconds: 238.52 ops/second Waiting for MDT destroys to complete - open/close 1463 (time 1746157770.92 total 10.01 last 146.16) - open/close 3053 (time 1746157780.92 total 20.01 last 158.93) - open/close 4555 (time 1746157790.92 total 30.02 last 150.16) - open/close 6103 (time 1746157800.92 total 40.02 last 154.79) - open/close 7544 (time 1746157810.93 total 50.02 last 144.06) - open/close 9432 (time 1746157820.93 total 60.02 last 188.79) - open/close 10000 (time 1746157824.29 total 63.38 last 169.06) - open/close 11854 (time 1746157834.29 total 73.38 last 185.38) - open/close 13604 (time 1746157844.29 total 83.38 last 174.97) - open/close 15161 (time 1746157854.29 total 93.38 last 155.69) - open/close 16643 (time 1746157864.29 total 103.39 last 148.14) - open/close 18347 (time 1746157874.30 total 113.39 last 170.33) - open/close 20000 (time 1746157882.47 total 121.57 last 202.25) - open/close 21840 (time 1746157892.47 total 131.57 last 183.99) - open/close 23462 (time 1746157902.47 total 141.57 last 162.16) - open/close 25152 (time 1746157912.48 total 151.57 last 168.93) - open/close 26851 (time 1746157922.48 total 161.57 last 169.90) - open/close 28597 (time 1746157932.48 total 171.58 last 174.50) - open/close 30000 (time 1746157941.78 total 180.87 last 150.98) - open/close 31491 (time 1746157951.78 total 190.87 last 149.10) - open/close 32897 (time 1746157961.78 total 200.88 last 140.55) - open/close 34251 (time 1746157971.78 total 210.88 last 135.40) - open/close 35644 (time 1746157981.79 total 220.88 last 139.22) - open/close 37046 (time 1746157991.79 total 230.89 last 140.11) - open/close 38460 (time 1746158001.80 total 240.90 last 141.28) - open/close 40000 (time 1746158010.47 total 249.56 last 177.74) - open/close 41793 (time 1746158020.47 total 259.56 last 179.29) - open/close 43657 (time 1746158030.47 total 269.56 last 186.37) - open/close 45323 (time 1746158040.47 total 279.56 last 166.59) - open/close 47080 (time 1746158050.47 total 289.57 last 175.63) - open/close 48826 (time 1746158060.48 total 299.57 last 174.56) - open/close 50000 (time 1746158067.51 total 306.60 last 166.88) - open/close 51401 (time 1746158077.51 total 316.61 last 140.07) - open/close 52892 (time 1746158087.52 total 326.61 last 149.04) - open/close 54246 (time 1746158097.52 total 336.61 last 135.39) - open/close 55574 (time 1746158107.52 total 346.62 last 132.75) - open/close 57057 (time 1746158117.53 total 356.62 last 148.20) - open/close 58876 (time 1746158127.53 total 366.62 last 181.86) - open/close 60000 (time 1746158133.67 total 372.76 last 183.19) - open/close 61723 (time 1746158143.67 total 382.76 last 172.26) - open/close 63261 (time 1746158153.67 total 392.76 last 153.79) total: 64768 open/close in 401.89 seconds: 161.16 ops/second - /unlink 2520 (time 1746158174.71 total 10.00 last 252.00) - /unlink 5102 (time 1746158184.71 total 20.00 last 258.17) - /unlink 7451 (time 1746158194.72 total 30.00 last 234.88) - /unlink 10000 (time 1746158203.72 total 39.01 last 283.00) - /unlink 12750 (time 1746158213.72 total 49.01 last 274.94) - /unlink 16062 (time 1746158223.73 total 59.01 last 331.09) - /unlink 19007 (time 1746158233.73 total 69.02 last 294.44) - /unlink 20000 (time 1746158238.04 total 73.33 last 230.32) - /unlink 22097 (time 1746158248.05 total 83.33 last 209.61) - /unlink 24418 (time 1746158258.05 total 93.33 last 232.05) - /unlink 27027 (time 1746158268.05 total 103.34 last 260.83) - /unlink 29802 (time 1746158278.05 total 113.34 last 277.43) - /unlink 32272 (time 1746158288.06 total 123.35 last 246.88) - /unlink 34737 (time 1746158298.06 total 133.35 last 246.48) - /unlink 37252 (time 1746158308.06 total 143.35 last 251.47) - /unlink 40000 (time 1746158317.88 total 153.16 last 279.92) - /unlink 42824 (time 1746158327.88 total 163.16 last 282.37) - /unlink 45544 (time 1746158337.88 total 173.17 last 271.98) - /unlink 48374 (time 1746158347.88 total 183.17 last 282.96) - /unlink 50000 (time 1746158354.24 total 189.52 last 255.74) - /unlink 52402 (time 1746158364.24 total 199.53 last 240.18) - /unlink 54901 (time 1746158374.24 total 209.53 last 249.82) - /unlink 57465 (time 1746158384.24 total 219.53 last 256.39) - /unlink 60000 (time 1746158393.94 total 229.23 last 261.34) - /unlink 62374 (time 1746158403.94 total 239.23 last 237.35) total: 64768 /unlink in 248.75 seconds: 260.38 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 PASS 106 (3939s) Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4072 sec ========== 00:01:23 (1746158483) === conf-sanity: start cleanup 00:01:24 (1746158484) === === conf-sanity: finish cleanup 00:01:25 (1746158485) ===