-----============= acceptance-small: conf-sanity ============----- Wed May 28 17:49:43 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg416-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg416-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg416-server oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg416-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg416-server: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg416-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:50:34 (1748469034) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg416-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg416-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5520 (time 1748469068.16 total 10.00 last 551.97) - open/close 10000 (time 1748469076.04 total 17.88 last 568.77) - open/close 15781 (time 1748469086.04 total 27.88 last 578.04) - open/close 20000 (time 1748469093.48 total 35.32 last 566.85) - open/close 25609 (time 1748469103.48 total 45.32 last 560.81) - open/close 30000 (time 1748469111.29 total 53.13 last 562.43) - open/close 35687 (time 1748469121.29 total 63.13 last 568.64) - open/close 40000 (time 1748469129.00 total 70.84 last 559.69) - open/close 45392 (time 1748469139.00 total 80.84 last 539.16) - open/close 50000 (time 1748469147.63 total 89.47 last 534.07) - open/close 55335 (time 1748469157.63 total 99.47 last 533.41) - open/close 60000 (time 1748469166.25 total 108.09 last 541.22) total: 64768 open/close in 116.81 seconds: 554.49 ops/second - /unlink 9303 (time 1748469185.79 total 10.00 last 930.28) - /unlink 18504 (time 1748469195.80 total 20.00 last 920.03) - /unlink 27605 (time 1748469205.80 total 30.00 last 910.06) - /unlink 30000 (time 1748469208.29 total 32.49 last 960.55) - /unlink 38959 (time 1748469218.29 total 42.50 last 895.83) - /unlink 47963 (time 1748469228.29 total 52.50 last 900.32) - /unlink 50000 (time 1748469230.65 total 54.85 last 863.82) - /unlink 59183 (time 1748469240.65 total 64.85 last 918.27) total: 64768 /unlink in 70.90 seconds: 913.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5609 (time 1748469282.02 total 10.00 last 560.84) - open/close 10000 (time 1748469290.02 total 17.99 last 549.41) - open/close 15308 (time 1748469300.02 total 27.99 last 530.80) - open/close 20000 (time 1748469308.71 total 36.69 last 539.46) - open/close 25399 (time 1748469318.71 total 46.69 last 539.84) - open/close 30000 (time 1748469327.68 total 55.66 last 513.26) - open/close 35239 (time 1748469337.68 total 65.66 last 523.87) - open/close 40000 (time 1748469346.65 total 74.62 last 531.03) - open/close 45285 (time 1748469356.65 total 84.62 last 528.40) - open/close 50000 (time 1748469365.72 total 93.70 last 519.62) - open/close 55369 (time 1748469375.72 total 103.70 last 536.90) - open/close 60000 (time 1748469384.61 total 112.58 last 521.13) total: 64768 open/close in 121.56 seconds: 532.79 ops/second - /unlink 9150 (time 1748469404.40 total 10.00 last 914.91) - /unlink 18399 (time 1748469414.40 total 20.00 last 924.90) - /unlink 27669 (time 1748469424.40 total 30.00 last 926.98) - /unlink 30000 (time 1748469427.00 total 32.60 last 896.59) - /unlink 39281 (time 1748469437.01 total 42.60 last 928.07) - /unlink 48487 (time 1748469447.01 total 52.60 last 920.53) - /unlink 57699 (time 1748469457.01 total 62.60 last 921.17) - /unlink 60000 (time 1748469459.36 total 64.96 last 976.64) total: 64768 /unlink in 69.98 seconds: 925.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5799 (time 1748469502.50 total 10.00 last 579.83) - open/close 10000 (time 1748469509.78 total 17.28 last 577.26) - open/close 15646 (time 1748469519.78 total 27.28 last 564.58) - open/close 20000 (time 1748469527.47 total 34.97 last 565.99) - open/close 25677 (time 1748469537.47 total 44.97 last 567.65) - open/close 30000 (time 1748469545.26 total 52.76 last 554.89) - open/close 35492 (time 1748469555.26 total 62.76 last 549.13) - open/close 40000 (time 1748469563.31 total 70.81 last 560.21) - open/close 45475 (time 1748469573.31 total 80.81 last 547.48) - open/close 50000 (time 1748469582.33 total 89.84 last 501.41) - open/close 55290 (time 1748469592.33 total 99.84 last 528.99) - open/close 60000 (time 1748469601.39 total 108.89 last 520.01) total: 64768 open/close in 117.66 seconds: 550.47 ops/second - /unlink 8948 (time 1748469620.98 total 10.00 last 894.63) - /unlink 17585 (time 1748469630.99 total 20.00 last 863.57) - /unlink 20000 (time 1748469633.53 total 22.54 last 950.35) - /unlink 29041 (time 1748469643.53 total 32.54 last 904.09) - /unlink 37440 (time 1748469653.53 total 42.55 last 839.87) - /unlink 40000 (time 1748469656.35 total 45.37 last 905.76) - /unlink 49332 (time 1748469666.35 total 55.37 last 933.18) - /unlink 58584 (time 1748469676.35 total 65.37 last 925.20) total: 64768 /unlink in 71.97 seconds: 899.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4355 (time 1748469717.65 total 10.00 last 435.41) - open/close 8622 (time 1748469727.66 total 20.00 last 426.63) - open/close 10000 (time 1748469731.18 total 23.53 last 390.86) - open/close 14289 (time 1748469741.18 total 33.53 last 428.89) - open/close 18547 (time 1748469751.18 total 43.53 last 425.78) - open/close 20000 (time 1748469754.77 total 47.12 last 404.51) - open/close 23178 (time 1748469764.78 total 57.13 last 317.71) - open/close 27212 (time 1748469774.78 total 67.13 last 403.39) - open/close 30000 (time 1748469784.09 total 76.43 last 299.50) - open/close 33213 (time 1748469794.09 total 86.44 last 321.23) - open/close 36089 (time 1748469804.09 total 96.44 last 287.58) - open/close 39337 (time 1748469814.09 total 106.44 last 324.77) - open/close 42313 (time 1748469824.13 total 116.47 last 296.52) - open/close 45209 (time 1748469834.13 total 126.48 last 289.56) - open/close 48271 (time 1748469844.13 total 136.48 last 306.09) - open/close 50000 (time 1748469850.82 total 143.17 last 258.39) - open/close 52713 (time 1748469860.82 total 153.17 last 271.28) - open/close 55517 (time 1748469870.82 total 163.17 last 280.39) - open/close 59244 (time 1748469880.82 total 173.17 last 372.69) - open/close 64359 (time 1748469890.83 total 183.17 last 511.45) total: 64768 open/close in 183.98 seconds: 352.04 ops/second - /unlink 9341 (time 1748469902.42 total 10.00 last 934.04) - /unlink 18545 (time 1748469912.42 total 20.00 last 920.37) - /unlink 25540 (time 1748469922.42 total 30.00 last 699.47) - /unlink 30000 (time 1748469931.01 total 38.59 last 519.53) - /unlink 35012 (time 1748469941.01 total 48.59 last 501.13) - /unlink 39945 (time 1748469951.01 total 58.59 last 493.28) - /unlink 45349 (time 1748469961.01 total 68.59 last 540.32) - /unlink 50000 (time 1748469969.84 total 77.42 last 526.92) - /unlink 55342 (time 1748469979.84 total 87.42 last 534.09) - /unlink 60000 (time 1748469988.41 total 95.99 last 543.41) total: 64768 /unlink in 104.83 seconds: 617.82 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3278 (time 1748470023.26 total 10.00 last 327.78) - open/close 6639 (time 1748470033.26 total 20.00 last 336.04) - open/close 10000 (time 1748470042.37 total 29.12 last 368.81) - open/close 12793 (time 1748470052.37 total 39.12 last 279.20) - open/close 15457 (time 1748470062.38 total 49.12 last 266.31) - open/close 18215 (time 1748470072.38 total 59.13 last 275.73) - open/close 20000 (time 1748470078.76 total 65.50 last 279.83) - open/close 23026 (time 1748470088.76 total 75.51 last 302.51) - open/close 25718 (time 1748470098.76 total 85.51 last 269.18) - open/close 28392 (time 1748470108.77 total 95.51 last 267.29) - open/close 30000 (time 1748470114.61 total 101.35 last 275.44) - open/close 32638 (time 1748470124.61 total 111.35 last 263.70) - open/close 35359 (time 1748470134.61 total 121.36 last 272.03) - open/close 39130 (time 1748470144.61 total 131.36 last 377.01) - open/close 40000 (time 1748470148.02 total 134.76 last 255.57) - open/close 42726 (time 1748470158.02 total 144.77 last 272.53) - open/close 45530 (time 1748470168.02 total 154.77 last 280.31) - open/close 48292 (time 1748470178.02 total 164.77 last 276.19) - open/close 50000 (time 1748470183.94 total 170.68 last 288.77) - open/close 53236 (time 1748470193.94 total 180.69 last 323.53) - open/close 55957 (time 1748470203.94 total 190.69 last 272.01) - open/close 58496 (time 1748470213.95 total 200.69 last 253.82) - open/close 60000 (time 1748470219.85 total 206.59 last 254.81) - open/close 62634 (time 1748470229.85 total 216.60 last 263.35) total: 64768 open/close in 224.21 seconds: 288.87 ops/second - /unlink 6691 (time 1748470248.74 total 10.00 last 669.09) - /unlink 10000 (time 1748470252.35 total 13.61 last 917.33) - /unlink 16933 (time 1748470262.35 total 23.61 last 693.28) - /unlink 20000 (time 1748470265.93 total 27.19 last 856.78) - /unlink 27744 (time 1748470275.93 total 37.19 last 774.40) - /unlink 30000 (time 1748470278.62 total 39.87 last 839.96) - /unlink 35328 (time 1748470288.62 total 49.87 last 532.77) - /unlink 40000 (time 1748470297.22 total 58.48 last 542.79) - /unlink 45547 (time 1748470307.23 total 68.48 last 554.63) - /unlink 50000 (time 1748470315.84 total 77.09 last 517.18) - /unlink 55093 (time 1748470325.84 total 87.09 last 509.29) - /unlink 60000 (time 1748470333.46 total 94.72 last 643.27) total: 64768 /unlink in 102.08 seconds: 634.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server unloading modules via unload_modules_local on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 PASS 106 (1347s) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: MODOPTS_LIBCFS= oleg416-server: Force libcfs to create 2 CPU partitions oleg416-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg416-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg416-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1420 sec ========== 18:13:24 (1748470404) === conf-sanity: start cleanup 18:13:25 (1748470405) === === conf-sanity: finish cleanup 18:13:26 (1748470406) ===