-----============= acceptance-small: conf-sanity ============----- Tue Jun 10 21:31:31 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg150-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg150-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg150-server oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: MODOPTS_LIBCFS= oleg150-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg150-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg150-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg150-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg150-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:33:02 (1749605582) Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: MODOPTS_LIBCFS= oleg150-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg150-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg150-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg150-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4232 (time 1749605627.46 total 10.00 last 423.14) - open/close 7965 (time 1749605637.46 total 20.00 last 373.19) - open/close 10000 (time 1749605642.07 total 24.62 last 440.94) - open/close 14769 (time 1749605652.08 total 34.62 last 476.78) - open/close 18140 (time 1749605662.08 total 44.62 last 337.02) - open/close 20000 (time 1749605667.21 total 49.76 last 362.19) - open/close 24163 (time 1749605677.21 total 59.76 last 416.28) - open/close 28047 (time 1749605687.21 total 69.76 last 388.36) - open/close 30000 (time 1749605692.94 total 75.49 last 341.01) - open/close 33061 (time 1749605702.94 total 85.49 last 306.09) - open/close 35922 (time 1749605712.94 total 95.49 last 286.08) - open/close 38923 (time 1749605722.95 total 105.49 last 300.04) - open/close 40000 (time 1749605726.83 total 109.38 last 277.04) - open/close 43349 (time 1749605736.83 total 119.38 last 334.84) - open/close 46175 (time 1749605746.84 total 129.38 last 282.50) - open/close 49886 (time 1749605756.84 total 139.39 last 371.04) - open/close 53563 (time 1749605766.84 total 149.39 last 367.69) - open/close 57476 (time 1749605776.84 total 159.39 last 391.25) - open/close 60000 (time 1749605782.37 total 164.92 last 456.61) total: 64768 open/close in 174.82 seconds: 370.48 ops/second - /unlink 8630 (time 1749605803.14 total 10.00 last 862.91) - /unlink 17605 (time 1749605813.14 total 20.00 last 897.47) - /unlink 20000 (time 1749605815.72 total 22.58 last 930.00) - /unlink 29024 (time 1749605825.72 total 32.58 last 902.40) - /unlink 38810 (time 1749605835.72 total 42.58 last 978.56) - /unlink 47624 (time 1749605845.72 total 52.58 last 881.34) - /unlink 50000 (time 1749605848.47 total 55.33 last 862.27) - /unlink 58904 (time 1749605858.47 total 65.33 last 890.32) total: 64768 /unlink in 72.56 seconds: 892.67 ops/second Waiting for MDT destroys to complete - open/close 3120 (time 1749605965.53 total 10.00 last 311.92) - open/close 6830 (time 1749605975.53 total 20.00 last 370.97) - open/close 10000 (time 1749605983.48 total 27.96 last 398.31) - open/close 14620 (time 1749605993.49 total 37.96 last 461.96) - open/close 18024 (time 1749606003.49 total 47.96 last 340.37) - open/close 20000 (time 1749606009.96 total 54.43 last 305.44) - open/close 23434 (time 1749606019.96 total 64.43 last 343.35) - open/close 26418 (time 1749606029.96 total 74.44 last 298.38) - open/close 29047 (time 1749606039.96 total 84.44 last 262.82) - open/close 30000 (time 1749606043.54 total 88.01 last 266.51) - open/close 32988 (time 1749606053.54 total 98.02 last 298.70) - open/close 35865 (time 1749606063.54 total 108.02 last 287.64) - open/close 39156 (time 1749606073.54 total 118.02 last 329.04) - open/close 42361 (time 1749606083.55 total 128.02 last 320.43) - open/close 44818 (time 1749606093.55 total 138.02 last 245.69) - open/close 47505 (time 1749606103.55 total 148.02 last 268.70) - open/close 50000 (time 1749606111.17 total 155.65 last 327.17) - open/close 53489 (time 1749606121.17 total 165.65 last 348.88) - open/close 56811 (time 1749606131.18 total 175.65 last 332.12) - open/close 60000 (time 1749606141.16 total 185.64 last 319.43) - open/close 63779 (time 1749606151.16 total 195.64 last 377.82) total: 64768 open/close in 199.35 seconds: 324.89 ops/second - /unlink 7293 (time 1749606166.24 total 10.00 last 729.23) - /unlink 10000 (time 1749606171.24 total 15.00 last 541.81) - /unlink 18053 (time 1749606181.24 total 25.00 last 805.23) - /unlink 20000 (time 1749606184.22 total 27.98 last 653.74) - /unlink 27852 (time 1749606194.22 total 37.98 last 785.10) - /unlink 30000 (time 1749606198.64 total 42.39 last 486.29) - /unlink 35869 (time 1749606208.64 total 52.40 last 586.83) - /unlink 40000 (time 1749606215.16 total 58.91 last 633.85) - /unlink 48853 (time 1749606225.16 total 68.91 last 885.26) - /unlink 54586 (time 1749606235.16 total 78.91 last 573.26) - /unlink 59494 (time 1749606245.16 total 88.92 last 490.74) - /unlink 64694 (time 1749606255.16 total 98.92 last 519.94) total: 64768 /unlink in 99.08 seconds: 653.69 ops/second Waiting for MDT destroys to complete - open/close 2628 (time 1749606316.53 total 10.00 last 262.73) - open/close 5403 (time 1749606326.53 total 20.00 last 277.46) - open/close 8574 (time 1749606336.53 total 30.01 last 316.99) - open/close 10000 (time 1749606341.50 total 34.98 last 286.87) - open/close 12715 (time 1749606351.50 total 44.98 last 271.44) - open/close 15539 (time 1749606361.50 total 54.98 last 282.40) - open/close 18427 (time 1749606371.50 total 64.98 last 288.80) - open/close 20000 (time 1749606376.95 total 70.43 last 288.78) - open/close 23424 (time 1749606386.95 total 80.43 last 342.36) - open/close 26352 (time 1749606396.95 total 90.43 last 292.76) - open/close 29715 (time 1749606406.95 total 100.43 last 336.27) - open/close 33015 (time 1749606416.96 total 110.43 last 329.93) - open/close 36414 (time 1749606426.96 total 120.43 last 339.87) - open/close 39314 (time 1749606436.96 total 130.44 last 289.98) - open/close 40000 (time 1749606439.78 total 133.25 last 243.52) - open/close 42730 (time 1749606449.78 total 143.25 last 272.97) - open/close 45364 (time 1749606459.78 total 153.25 last 263.40) - open/close 47852 (time 1749606469.78 total 163.25 last 248.76) - open/close 50000 (time 1749606477.93 total 171.41 last 263.55) - open/close 52570 (time 1749606487.93 total 181.41 last 256.96) - open/close 55029 (time 1749606497.93 total 191.41 last 245.86) - open/close 58040 (time 1749606507.93 total 201.41 last 301.04) - open/close 60000 (time 1749606512.95 total 206.43 last 390.80) - open/close 63422 (time 1749606522.95 total 216.43 last 342.19) total: 64768 open/close in 220.90 seconds: 293.20 ops/second - /unlink 5231 (time 1749606538.68 total 10.00 last 523.05) - /unlink 10000 (time 1749606548.46 total 19.78 last 487.77) - /unlink 14478 (time 1749606558.46 total 29.78 last 447.70) - /unlink 19332 (time 1749606568.46 total 39.78 last 485.33) - /unlink 23814 (time 1749606578.47 total 49.78 last 448.17) - /unlink 28424 (time 1749606588.47 total 59.78 last 460.94) - /unlink 30000 (time 1749606591.98 total 63.30 last 447.97) - /unlink 34562 (time 1749606601.99 total 73.30 last 456.17) - /unlink 39177 (time 1749606611.99 total 83.30 last 461.50) - /unlink 43533 (time 1749606621.99 total 93.31 last 435.27) - /unlink 48915 (time 1749606631.99 total 103.31 last 538.14) - /unlink 54267 (time 1749606642.00 total 113.31 last 535.10) - /unlink 58779 (time 1749606652.00 total 123.31 last 451.17) - /unlink 60000 (time 1749606654.80 total 126.11 last 435.92) - /unlink 64528 (time 1749606664.80 total 136.12 last 452.71) total: 64768 /unlink in 136.64 seconds: 473.99 ops/second Waiting for MDT destroys to complete - open/close 2452 (time 1749606682.65 total 10.00 last 245.18) - open/close 5199 (time 1749606692.66 total 20.00 last 274.61) - open/close 7729 (time 1749606702.66 total 30.01 last 252.93) - open/close 10000 (time 1749606710.92 total 38.27 last 274.77) - open/close 12963 (time 1749606720.92 total 48.27 last 296.28) - open/close 15642 (time 1749606730.92 total 58.27 last 267.89) - open/close 18091 (time 1749606740.93 total 68.28 last 244.86) - open/close 20000 (time 1749606748.61 total 75.96 last 248.39) - open/close 22479 (time 1749606758.61 total 85.96 last 247.84) - open/close 25000 (time 1749606768.62 total 95.97 last 252.04) - open/close 27462 (time 1749606778.62 total 105.97 last 246.09) - open/close 29910 (time 1749606788.62 total 115.97 last 244.75) - open/close 32780 (time 1749606798.63 total 125.98 last 286.91) - open/close 36545 (time 1749606808.63 total 135.98 last 376.49) - open/close 39210 (time 1749606818.63 total 145.98 last 266.49) - open/close 44469 (time 1749606828.63 total 155.98 last 525.87) - open/close 49663 (time 1749606838.63 total 165.98 last 519.31) - open/close 53360 (time 1749606848.63 total 175.98 last 369.65) - open/close 58524 (time 1749606858.63 total 185.98 last 516.33) - open/close 60000 (time 1749606861.58 total 188.93 last 500.42) total: 64768 open/close in 198.28 seconds: 326.64 ops/second - /unlink 9008 (time 1749606881.73 total 10.00 last 900.79) - /unlink 18171 (time 1749606891.73 total 20.00 last 916.25) - /unlink 27539 (time 1749606901.73 total 30.00 last 936.73) - /unlink 30000 (time 1749606904.28 total 32.55 last 966.93) - /unlink 38137 (time 1749606914.28 total 42.55 last 813.53) - /unlink 40000 (time 1749606918.33 total 46.60 last 459.77) - /unlink 44968 (time 1749606928.33 total 56.60 last 496.79) - /unlink 50000 (time 1749606936.51 total 64.78 last 615.57) - /unlink 58746 (time 1749606946.51 total 74.78 last 874.54) total: 64768 /unlink in 80.94 seconds: 800.22 ops/second Waiting for MDT destroys to complete - open/close 2786 (time 1749607041.99 total 10.00 last 278.57) - open/close 6254 (time 1749607051.99 total 20.00 last 346.75) - open/close 10000 (time 1749607061.97 total 29.98 last 375.30) - open/close 14316 (time 1749607071.97 total 39.98 last 431.57) - open/close 17448 (time 1749607081.97 total 49.99 last 313.11) - open/close 20000 (time 1749607090.32 total 58.34 last 305.65) - open/close 22587 (time 1749607100.32 total 68.34 last 258.63) - open/close 24970 (time 1749607110.32 total 78.34 last 238.28) - open/close 29680 (time 1749607120.32 total 88.34 last 470.99) - open/close 33113 (time 1749607130.33 total 98.34 last 343.19) - open/close 35731 (time 1749607140.33 total 108.34 last 261.78) - open/close 38798 (time 1749607150.33 total 118.35 last 306.67) - open/close 40000 (time 1749607152.60 total 120.62 last 528.73) - open/close 44474 (time 1749607162.60 total 130.62 last 447.35) - open/close 46915 (time 1749607172.61 total 140.62 last 244.03) - open/close 49259 (time 1749607182.61 total 150.62 last 234.38) - open/close 50000 (time 1749607185.39 total 153.41 last 265.91) - open/close 52389 (time 1749607195.40 total 163.41 last 238.88) - open/close 55397 (time 1749607205.40 total 173.41 last 300.76) - open/close 59123 (time 1749607215.40 total 183.42 last 372.50) - open/close 60000 (time 1749607218.36 total 186.38 last 295.97) - open/close 62394 (time 1749607228.36 total 196.38 last 239.37) total: 64768 open/close in 206.31 seconds: 313.93 ops/second - /unlink 4314 (time 1749607249.62 total 10.00 last 431.35) - /unlink 8547 (time 1749607259.62 total 20.00 last 423.28) - /unlink 10000 (time 1749607262.91 total 23.30 last 441.15) - /unlink 14183 (time 1749607272.91 total 33.30 last 418.24) - /unlink 18413 (time 1749607282.91 total 43.30 last 422.98) - /unlink 20000 (time 1749607286.56 total 46.94 last 435.47) - /unlink 24245 (time 1749607296.56 total 56.94 last 424.40) - /unlink 28930 (time 1749607306.56 total 66.95 last 468.42) - /unlink 30000 (time 1749607309.05 total 69.43 last 430.66) - /unlink 35329 (time 1749607319.05 total 79.43 last 532.88) - /unlink 39580 (time 1749607329.05 total 89.43 last 425.05) - /unlink 44068 (time 1749607339.05 total 99.43 last 448.73) - /unlink 48373 (time 1749607349.05 total 109.43 last 430.44) - /unlink 50000 (time 1749607352.87 total 113.25 last 426.58) - /unlink 54318 (time 1749607362.87 total 123.25 last 431.74) - /unlink 58767 (time 1749607372.87 total 133.25 last 444.82) - /unlink 60000 (time 1749607375.67 total 136.05 last 440.66) total: 64768 /unlink in 144.63 seconds: 447.82 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server unloading modules via unload_modules_local on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 PASS 106 (1834s) Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: MODOPTS_LIBCFS= oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg150-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1951 sec ========== 22:04:02 (1749607442) === conf-sanity: start cleanup 22:04:03 (1749607443) === === conf-sanity: finish cleanup 22:04:04 (1749607444) ===