-----============= acceptance-small: conf-sanity ============----- Thu May 1 11:07:34 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg250-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg250-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg250-server oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg250-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg250-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg250-server: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg250-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg250-server: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg250-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:09:00 (1746112140) Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg250-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg250-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 336 65472 1% /mnt/lustre[OST:0] filesystem_summary: 66001 529 65472 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5464 (time 1746112180.60 total 10.00 last 546.33) - open/close 10000 (time 1746112189.22 total 18.62 last 526.17) - open/close 15472 (time 1746112199.22 total 28.62 last 547.19) - open/close 20000 (time 1746112207.20 total 36.60 last 567.54) - open/close 25679 (time 1746112217.20 total 46.60 last 567.90) - open/close 30000 (time 1746112224.90 total 54.30 last 561.05) - open/close 35763 (time 1746112234.90 total 64.30 last 576.21) - open/close 40000 (time 1746112242.60 total 72.00 last 550.48) - open/close 45566 (time 1746112252.60 total 82.00 last 556.51) - open/close 50000 (time 1746112260.78 total 90.18 last 542.15) - open/close 55349 (time 1746112270.78 total 100.18 last 534.80) - open/close 60000 (time 1746112279.63 total 109.03 last 525.62) total: 64768 open/close in 118.05 seconds: 548.63 ops/second - /unlink 9035 (time 1746112299.59 total 10.00 last 903.47) - /unlink 18163 (time 1746112309.59 total 20.00 last 912.75) - /unlink 20000 (time 1746112311.65 total 22.06 last 891.75) - /unlink 28675 (time 1746112321.65 total 32.06 last 867.45) - /unlink 37589 (time 1746112331.65 total 42.06 last 891.31) - /unlink 40000 (time 1746112334.39 total 44.81 last 878.34) - /unlink 50000 (time 1746112344.30 total 54.72 last 1009.03) - /unlink 60000 (time 1746112353.19 total 63.61 last 1124.82) total: 64768 /unlink in 67.62 seconds: 957.86 ops/second Waiting for MDT destroys to complete - open/close 5588 (time 1746112459.22 total 10.00 last 558.78) - open/close 10000 (time 1746112467.39 total 18.17 last 539.91) - open/close 15502 (time 1746112477.40 total 28.17 last 550.13) - open/close 20000 (time 1746112485.50 total 36.28 last 555.17) - open/close 25624 (time 1746112495.50 total 46.28 last 562.39) - open/close 30000 (time 1746112503.40 total 54.17 last 554.08) - open/close 35403 (time 1746112513.40 total 64.17 last 540.21) - open/close 40000 (time 1746112521.78 total 72.56 last 548.52) - open/close 45555 (time 1746112531.78 total 82.56 last 555.49) - open/close 50000 (time 1746112539.96 total 90.74 last 543.15) - open/close 55471 (time 1746112549.96 total 100.74 last 547.09) - open/close 60000 (time 1746112558.30 total 109.07 last 543.41) total: 64768 open/close in 118.07 seconds: 548.56 ops/second - /unlink 9567 (time 1746112578.14 total 10.00 last 956.63) - /unlink 19487 (time 1746112588.14 total 20.00 last 991.93) - /unlink 29156 (time 1746112598.14 total 30.00 last 966.83) - /unlink 39130 (time 1746112608.14 total 40.00 last 997.35) - /unlink 49261 (time 1746112618.14 total 50.00 last 1013.09) - /unlink 59363 (time 1746112628.14 total 60.00 last 1010.12) total: 64768 /unlink in 65.35 seconds: 991.10 ops/second Waiting for MDT destroys to complete - open/close 5787 (time 1746112740.39 total 10.00 last 578.68) - open/close 10000 (time 1746112748.42 total 18.03 last 524.85) - open/close 14187 (time 1746112758.42 total 28.03 last 418.68) - open/close 19439 (time 1746112768.42 total 38.03 last 525.13) - open/close 24670 (time 1746112778.42 total 48.03 last 523.07) - open/close 29947 (time 1746112788.42 total 58.03 last 527.69) - open/close 35218 (time 1746112798.42 total 68.03 last 527.10) - open/close 40000 (time 1746112807.85 total 77.46 last 507.14) - open/close 45229 (time 1746112817.85 total 87.46 last 522.87) - open/close 50000 (time 1746112827.10 total 96.71 last 515.85) - open/close 55155 (time 1746112837.10 total 106.71 last 515.43) - open/close 60000 (time 1746112846.71 total 116.31 last 504.64) - open/close 64308 (time 1746112856.71 total 126.31 last 430.73) total: 64768 open/close in 127.72 seconds: 507.10 ops/second - /unlink 8201 (time 1746112869.38 total 10.00 last 819.75) - /unlink 10000 (time 1746112871.46 total 12.09 last 863.67) - /unlink 17110 (time 1746112881.46 total 22.09 last 710.98) - /unlink 20000 (time 1746112884.70 total 25.32 last 894.54) - /unlink 26364 (time 1746112894.70 total 35.32 last 636.34) - /unlink 30000 (time 1746112899.43 total 40.06 last 767.62) - /unlink 36442 (time 1746112909.43 total 50.06 last 644.13) - /unlink 40000 (time 1746112914.70 total 55.33 last 675.33) - /unlink 49013 (time 1746112924.70 total 65.33 last 901.26) - /unlink 57102 (time 1746112934.70 total 75.33 last 808.77) - /unlink 60000 (time 1746112938.93 total 79.56 last 685.14) total: 64768 /unlink in 84.43 seconds: 767.14 ops/second Waiting for MDT destroys to complete - open/close 4515 (time 1746113031.27 total 10.00 last 451.45) - open/close 8900 (time 1746113041.27 total 20.00 last 438.49) - open/close 10000 (time 1746113043.75 total 22.48 last 443.78) - open/close 14437 (time 1746113053.75 total 32.48 last 443.67) - open/close 18557 (time 1746113063.75 total 42.48 last 411.93) - open/close 20000 (time 1746113067.77 total 46.50 last 359.35) - open/close 23709 (time 1746113077.77 total 56.50 last 370.83) - open/close 27414 (time 1746113087.77 total 66.50 last 370.49) - open/close 30000 (time 1746113096.67 total 75.40 last 290.67) - open/close 33869 (time 1746113106.67 total 85.40 last 386.89) - open/close 37407 (time 1746113116.67 total 95.40 last 353.79) - open/close 40000 (time 1746113122.72 total 101.45 last 428.66) - open/close 43854 (time 1746113132.72 total 111.45 last 385.32) - open/close 47545 (time 1746113142.72 total 121.45 last 369.07) - open/close 50000 (time 1746113149.37 total 128.10 last 369.35) - open/close 53242 (time 1746113159.37 total 138.10 last 324.10) - open/close 56330 (time 1746113169.37 total 148.10 last 308.68) - open/close 59419 (time 1746113179.37 total 158.10 last 308.88) - open/close 60000 (time 1746113181.50 total 160.23 last 273.40) - open/close 63166 (time 1746113191.50 total 170.23 last 316.58) total: 64768 open/close in 175.27 seconds: 369.52 ops/second - /unlink 5873 (time 1746113207.82 total 10.00 last 587.29) - /unlink 10000 (time 1746113216.07 total 18.25 last 499.97) - /unlink 15637 (time 1746113226.08 total 28.26 last 563.35) - /unlink 20000 (time 1746113234.05 total 36.23 last 547.57) - /unlink 27013 (time 1746113244.05 total 46.23 last 701.23) - /unlink 30000 (time 1746113248.71 total 50.89 last 641.38) - /unlink 35519 (time 1746113258.71 total 60.89 last 551.80) - /unlink 40000 (time 1746113266.65 total 68.83 last 564.24) - /unlink 45842 (time 1746113276.65 total 78.83 last 584.20) - /unlink 50000 (time 1746113282.42 total 84.61 last 720.00) - /unlink 57110 (time 1746113292.42 total 94.61 last 710.96) - /unlink 60000 (time 1746113295.41 total 97.59 last 968.12) total: 64768 /unlink in 102.88 seconds: 629.55 ops/second Waiting for MDT destroys to complete - open/close 3276 (time 1746113366.76 total 10.00 last 327.54) - open/close 7226 (time 1746113376.76 total 20.00 last 394.90) - open/close 10000 (time 1746113383.40 total 26.64 last 418.06) - open/close 13428 (time 1746113393.40 total 36.64 last 342.79) - open/close 17056 (time 1746113403.40 total 46.64 last 362.74) - open/close 20000 (time 1746113410.84 total 54.08 last 395.85) - open/close 23590 (time 1746113420.84 total 64.08 last 358.99) - open/close 27250 (time 1746113430.84 total 74.08 last 365.99) - open/close 30000 (time 1746113437.01 total 80.25 last 445.68) - open/close 34028 (time 1746113447.01 total 90.25 last 402.73) - open/close 38618 (time 1746113457.01 total 100.25 last 458.99) - open/close 40000 (time 1746113460.29 total 103.53 last 421.38) - open/close 43447 (time 1746113470.29 total 113.53 last 344.65) - open/close 46850 (time 1746113480.29 total 123.53 last 340.28) - open/close 50000 (time 1746113489.22 total 132.46 last 352.79) - open/close 53986 (time 1746113499.22 total 142.47 last 398.46) - open/close 56652 (time 1746113509.22 total 152.47 last 266.59) - open/close 59178 (time 1746113519.23 total 162.47 last 252.52) - open/close 60000 (time 1746113522.20 total 165.45 last 276.12) - open/close 62987 (time 1746113532.21 total 175.45 last 298.63) total: 64768 open/close in 181.55 seconds: 356.75 ops/second - /unlink 5660 (time 1746113549.43 total 10.00 last 565.94) - /unlink 10000 (time 1746113557.35 total 17.92 last 548.23) - /unlink 15190 (time 1746113567.35 total 27.92 last 518.92) - /unlink 20000 (time 1746113574.21 total 34.78 last 701.31) - /unlink 25001 (time 1746113584.21 total 44.78 last 500.01) - /unlink 30000 (time 1746113593.59 total 54.16 last 532.68) - /unlink 36498 (time 1746113603.59 total 64.16 last 649.76) - /unlink 40000 (time 1746113609.20 total 69.77 last 625.24) - /unlink 47098 (time 1746113619.20 total 79.77 last 709.78) - /unlink 50000 (time 1746113623.12 total 83.69 last 739.37) - /unlink 55506 (time 1746113633.12 total 93.69 last 550.54) - /unlink 60000 (time 1746113642.46 total 103.03 last 481.34) total: 64768 /unlink in 111.00 seconds: 583.47 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server unloading modules via unload_modules_local on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 PASS 106 (1575s) Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg250-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg250-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1685 sec ========== 11:35:39 (1746113739) === conf-sanity: start cleanup 11:35:40 (1746113740) === === conf-sanity: finish cleanup 11:35:41 (1746113741) ===