-----============= acceptance-small: conf-sanity ============----- Fri Aug 9 21:17:28 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg455-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg455-server oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg455-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg455-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg455-server: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg455-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:18:22 (1723252702) Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg455-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg455-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5755 (time 1723252736.00 total 10.00 last 575.46) - open/close 10000 (time 1723252744.70 total 18.70 last 487.94) - open/close 11758 (time 1723252754.70 total 28.71 last 175.70) - open/close 13335 (time 1723252764.70 total 38.71 last 157.69) - open/close 15568 (time 1723252774.71 total 48.71 last 223.23) - open/close 19004 (time 1723252784.71 total 58.71 last 343.56) - open/close 20000 (time 1723252788.16 total 62.16 last 288.88) - open/close 23347 (time 1723252798.16 total 72.16 last 334.61) - open/close 26571 (time 1723252808.16 total 82.16 last 322.33) - open/close 29711 (time 1723252818.16 total 92.17 last 313.94) - open/close 34087 (time 1723252828.16 total 102.17 last 437.59) - open/close 38027 (time 1723252838.16 total 112.17 last 393.99) - open/close 40000 (time 1723252844.69 total 118.70 last 302.14) - open/close 42931 (time 1723252854.70 total 128.70 last 293.04) - open/close 45293 (time 1723252864.70 total 138.70 last 236.18) - open/close 48122 (time 1723252874.70 total 148.70 last 282.85) - open/close 50000 (time 1723252880.61 total 154.61 last 317.71) - open/close 53359 (time 1723252890.61 total 164.61 last 335.85) - open/close 56622 (time 1723252900.61 total 174.62 last 326.24) - open/close 60000 (time 1723252910.54 total 184.54 last 340.35) - open/close 63524 (time 1723252920.54 total 194.54 last 352.38) total: 64768 open/close in 198.15 seconds: 326.87 ops/second - /unlink 6101 (time 1723252935.17 total 10.00 last 610.00) - /unlink 10000 (time 1723252942.18 total 17.01 last 556.16) - /unlink 14977 (time 1723252952.19 total 27.01 last 497.56) - /unlink 19346 (time 1723252962.19 total 37.02 last 436.82) - /unlink 23415 (time 1723252972.19 total 47.02 last 406.87) - /unlink 28443 (time 1723252982.19 total 57.02 last 502.75) - /unlink 30000 (time 1723252984.98 total 59.81 last 557.02) - /unlink 36190 (time 1723252994.99 total 69.82 last 618.92) - /unlink 40000 (time 1723253002.18 total 77.01 last 529.38) - /unlink 44882 (time 1723253012.18 total 87.01 last 488.17) - /unlink 49443 (time 1723253022.19 total 97.02 last 455.97) - /unlink 52739 (time 1723253032.19 total 107.02 last 329.54) - /unlink 56714 (time 1723253042.19 total 117.02 last 397.44) - /unlink 60000 (time 1723253050.57 total 125.40 last 392.10) - /unlink 64400 (time 1723253060.57 total 135.40 last 439.98) total: 64768 /unlink in 136.12 seconds: 475.80 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2603 (time 1723253089.59 total 10.00 last 260.27) - open/close 5034 (time 1723253099.59 total 20.00 last 243.04) - open/close 7461 (time 1723253109.59 total 30.01 last 242.66) - open/close 10000 (time 1723253118.23 total 38.64 last 293.93) - open/close 12797 (time 1723253128.23 total 48.65 last 279.62) - open/close 15377 (time 1723253138.24 total 58.65 last 257.96) - open/close 17995 (time 1723253148.24 total 68.65 last 261.75) - open/close 20000 (time 1723253154.89 total 75.31 last 301.19) - open/close 22763 (time 1723253164.89 total 85.31 last 276.29) - open/close 25726 (time 1723253174.90 total 95.31 last 296.20) - open/close 28413 (time 1723253184.90 total 105.31 last 268.61) - open/close 30000 (time 1723253189.97 total 110.38 last 313.32) - open/close 32732 (time 1723253199.97 total 120.38 last 273.17) - open/close 35118 (time 1723253209.97 total 130.38 last 238.53) - open/close 37304 (time 1723253219.97 total 140.38 last 218.58) - open/close 39578 (time 1723253229.97 total 150.39 last 227.33) - open/close 41837 (time 1723253239.98 total 160.39 last 225.86) - open/close 44046 (time 1723253249.98 total 170.39 last 220.84) - open/close 46260 (time 1723253259.98 total 180.39 last 221.35) - open/close 48487 (time 1723253269.98 total 190.39 last 222.68) - open/close 50000 (time 1723253277.34 total 197.75 last 205.72) - open/close 51988 (time 1723253287.34 total 207.75 last 198.78) - open/close 54529 (time 1723253297.34 total 217.75 last 254.08) - open/close 56764 (time 1723253307.34 total 227.75 last 223.46) - open/close 58996 (time 1723253317.34 total 237.75 last 223.20) - open/close 60000 (time 1723253321.31 total 241.72 last 253.08) - open/close 62507 (time 1723253331.31 total 251.72 last 250.68) total: 64768 open/close in 260.46 seconds: 248.66 ops/second - /unlink 4799 (time 1723253351.08 total 10.00 last 479.84) - /unlink 9637 (time 1723253361.08 total 20.00 last 483.79) - /unlink 14362 (time 1723253371.08 total 30.00 last 472.46) - /unlink 18914 (time 1723253381.08 total 40.00 last 455.15) - /unlink 20000 (time 1723253383.66 total 42.58 last 421.76) - /unlink 24205 (time 1723253393.66 total 52.58 last 420.35) - /unlink 29379 (time 1723253403.66 total 62.58 last 517.36) - /unlink 34858 (time 1723253413.66 total 72.58 last 547.82) - /unlink 40000 (time 1723253423.07 total 81.99 last 546.55) - /unlink 45499 (time 1723253433.07 total 91.99 last 549.82) - /unlink 50000 (time 1723253440.92 total 99.84 last 573.56) - /unlink 56197 (time 1723253450.92 total 109.84 last 619.69) - /unlink 60000 (time 1723253458.52 total 117.44 last 500.66) total: 64768 /unlink in 125.69 seconds: 515.31 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4256 (time 1723253494.86 total 10.00 last 425.54) - open/close 8410 (time 1723253504.86 total 20.00 last 415.38) - open/close 10000 (time 1723253508.98 total 24.12 last 386.28) - open/close 13488 (time 1723253518.98 total 34.12 last 348.72) - open/close 17475 (time 1723253528.98 total 44.12 last 398.65) - open/close 20000 (time 1723253534.40 total 49.54 last 466.24) - open/close 24349 (time 1723253544.40 total 59.54 last 434.90) - open/close 28239 (time 1723253554.40 total 69.54 last 388.94) - open/close 30000 (time 1723253559.16 total 74.30 last 370.05) - open/close 33869 (time 1723253569.16 total 84.30 last 386.89) - open/close 38266 (time 1723253579.16 total 94.30 last 439.65) - open/close 40000 (time 1723253583.45 total 98.59 last 403.79) - open/close 44134 (time 1723253593.46 total 108.59 last 413.35) - open/close 47367 (time 1723253603.46 total 118.60 last 323.24) - open/close 50000 (time 1723253610.65 total 125.79 last 366.26) - open/close 53868 (time 1723253620.65 total 135.79 last 386.79) - open/close 58706 (time 1723253630.65 total 145.79 last 483.77) - open/close 60000 (time 1723253633.46 total 148.60 last 460.36) total: 64768 open/close in 157.99 seconds: 409.94 ops/second - /unlink 8663 (time 1723253653.68 total 10.00 last 866.20) - /unlink 16032 (time 1723253663.69 total 20.00 last 736.77) - /unlink 20000 (time 1723253669.40 total 25.72 last 694.07) - /unlink 27292 (time 1723253679.40 total 35.72 last 729.15) - /unlink 30000 (time 1723253682.69 total 39.01 last 824.49) - /unlink 38055 (time 1723253692.69 total 49.01 last 805.40) - /unlink 40000 (time 1723253695.20 total 51.52 last 773.78) - /unlink 47905 (time 1723253705.20 total 61.52 last 790.42) - /unlink 50000 (time 1723253707.68 total 64.00 last 846.69) - /unlink 57562 (time 1723253717.68 total 74.00 last 756.15) - /unlink 60000 (time 1723253721.06 total 77.38 last 720.67) total: 64768 /unlink in 84.18 seconds: 769.40 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4365 (time 1723253756.40 total 10.00 last 436.48) - open/close 8019 (time 1723253766.40 total 20.00 last 365.32) - open/close 10000 (time 1723253771.21 total 24.81 last 412.24) - open/close 15040 (time 1723253781.21 total 34.81 last 503.95) - open/close 20000 (time 1723253790.99 total 44.59 last 507.07) - open/close 24614 (time 1723253800.99 total 54.59 last 461.30) - open/close 29276 (time 1723253810.99 total 64.59 last 466.17) - open/close 34101 (time 1723253820.99 total 74.60 last 482.42) - open/close 39454 (time 1723253831.00 total 84.60 last 535.27) - open/close 44730 (time 1723253841.00 total 94.60 last 527.56) - open/close 50000 (time 1723253850.76 total 104.36 last 539.92) - open/close 54950 (time 1723253860.76 total 114.36 last 494.93) - open/close 59391 (time 1723253870.76 total 124.36 last 444.01) - open/close 64109 (time 1723253880.76 total 134.36 last 471.70) total: 64768 open/close in 135.71 seconds: 477.25 ops/second - /unlink 8381 (time 1723253892.98 total 10.00 last 838.06) - /unlink 10000 (time 1723253895.11 total 12.14 last 757.91) - /unlink 17931 (time 1723253905.12 total 22.14 last 793.01) - /unlink 20000 (time 1723253907.35 total 24.37 last 927.02) - /unlink 29392 (time 1723253917.35 total 34.37 last 939.13) - /unlink 38502 (time 1723253927.35 total 44.37 last 910.91) - /unlink 47874 (time 1723253937.35 total 54.37 last 937.14) - /unlink 50000 (time 1723253939.67 total 56.69 last 916.22) - /unlink 58587 (time 1723253949.67 total 66.69 last 858.67) total: 64768 /unlink in 73.37 seconds: 882.76 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4229 (time 1723253991.68 total 10.00 last 422.85) - open/close 9304 (time 1723254001.68 total 20.00 last 507.43) - open/close 14612 (time 1723254011.68 total 30.00 last 530.78) - open/close 19902 (time 1723254021.68 total 40.00 last 528.97) - open/close 25475 (time 1723254031.68 total 50.00 last 557.24) - open/close 30000 (time 1723254040.35 total 58.67 last 522.19) - open/close 34701 (time 1723254050.35 total 68.67 last 470.04) - open/close 39570 (time 1723254060.35 total 78.67 last 486.84) - open/close 44456 (time 1723254070.35 total 88.67 last 488.55) - open/close 49276 (time 1723254080.35 total 98.67 last 481.96) - open/close 53847 (time 1723254090.35 total 108.67 last 457.08) - open/close 58584 (time 1723254100.35 total 118.68 last 473.67) - open/close 60000 (time 1723254103.25 total 121.57 last 489.24) - open/close 64730 (time 1723254113.25 total 131.57 last 472.92) total: 64768 open/close in 131.64 seconds: 492.01 ops/second - /unlink 8141 (time 1723254124.18 total 10.00 last 814.04) - /unlink 10000 (time 1723254126.52 total 12.34 last 793.98) - /unlink 18257 (time 1723254136.52 total 22.34 last 825.62) - /unlink 20000 (time 1723254138.62 total 24.44 last 832.45) - /unlink 28126 (time 1723254148.62 total 34.44 last 812.58) - /unlink 30000 (time 1723254150.82 total 36.64 last 850.49) - /unlink 38437 (time 1723254160.82 total 46.64 last 843.63) - /unlink 47014 (time 1723254170.82 total 56.64 last 857.63) - /unlink 50000 (time 1723254174.30 total 60.12 last 858.88) - /unlink 58560 (time 1723254184.30 total 70.12 last 855.94) total: 64768 /unlink in 77.40 seconds: 836.75 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server unloading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 PASS 106 (1533s) Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg455-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1605 sec ========== 21:44:13 (1723254253) === conf-sanity: start cleanup 21:44:13 (1723254253) === === conf-sanity: finish cleanup 21:44:13 (1723254253) ===