-----============= acceptance-small: conf-sanity ============----- Tue Mar 11 07:17:46 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg455-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 32b 32c 32e 32h 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg455-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg455-server oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg455-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg455-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg455-server: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg455-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:18:42 (1741691922) Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg455-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg455-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4768 (time 1741691955.82 total 10.00 last 476.79) - open/close 9075 (time 1741691965.82 total 20.00 last 430.57) - open/close 10000 (time 1741691967.85 total 22.03 last 456.07) - open/close 14822 (time 1741691977.85 total 32.03 last 482.18) - open/close 20000 (time 1741691986.62 total 40.80 last 590.67) - open/close 25845 (time 1741691996.62 total 50.80 last 584.40) - open/close 30000 (time 1741692004.33 total 58.51 last 538.98) - open/close 35910 (time 1741692014.33 total 68.51 last 590.91) - open/close 40000 (time 1741692021.93 total 76.12 last 537.79) - open/close 45386 (time 1741692031.94 total 86.12 last 538.51) - open/close 50000 (time 1741692041.00 total 95.18 last 509.19) - open/close 55204 (time 1741692051.00 total 105.18 last 520.36) - open/close 60000 (time 1741692060.13 total 114.32 last 524.88) total: 64768 open/close in 124.16 seconds: 521.67 ops/second - /unlink 8858 (time 1741692081.06 total 10.00 last 885.76) - /unlink 17939 (time 1741692091.06 total 20.00 last 908.02) - /unlink 20000 (time 1741692093.29 total 22.23 last 925.46) - /unlink 29093 (time 1741692103.29 total 32.23 last 909.22) - /unlink 38513 (time 1741692113.29 total 42.23 last 941.87) - /unlink 47811 (time 1741692123.29 total 52.23 last 929.70) - /unlink 50000 (time 1741692125.52 total 54.47 last 979.60) - /unlink 59518 (time 1741692135.52 total 64.47 last 951.76) total: 64768 /unlink in 69.93 seconds: 926.15 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5172 (time 1741692178.59 total 10.00 last 517.16) - open/close 10000 (time 1741692187.89 total 19.30 last 519.40) - open/close 15227 (time 1741692197.89 total 29.30 last 522.70) - open/close 20000 (time 1741692207.13 total 38.54 last 516.44) - open/close 25294 (time 1741692217.13 total 48.54 last 529.35) - open/close 30000 (time 1741692226.06 total 57.47 last 526.98) - open/close 35187 (time 1741692236.07 total 67.47 last 518.58) - open/close 40000 (time 1741692245.43 total 76.83 last 514.23) - open/close 45192 (time 1741692255.43 total 86.83 last 519.15) - open/close 50000 (time 1741692264.80 total 96.20 last 513.03) - open/close 55322 (time 1741692274.80 total 106.21 last 532.13) - open/close 60000 (time 1741692283.76 total 115.17 last 521.97) total: 64768 open/close in 124.28 seconds: 521.15 ops/second - /unlink 9245 (time 1741692303.70 total 10.00 last 924.42) - /unlink 18453 (time 1741692313.70 total 20.00 last 920.72) - /unlink 27343 (time 1741692323.70 total 30.00 last 888.91) - /unlink 30000 (time 1741692326.57 total 32.87 last 927.35) - /unlink 39072 (time 1741692336.57 total 42.87 last 907.19) - /unlink 48199 (time 1741692346.57 total 52.87 last 912.62) - /unlink 50000 (time 1741692348.58 total 54.89 last 893.04) - /unlink 59590 (time 1741692358.58 total 64.89 last 958.93) total: 64768 /unlink in 70.25 seconds: 921.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5430 (time 1741692403.87 total 10.00 last 542.91) - open/close 10000 (time 1741692412.18 total 18.32 last 549.67) - open/close 15541 (time 1741692422.18 total 28.32 last 554.07) - open/close 20000 (time 1741692430.38 total 36.51 last 543.92) - open/close 25421 (time 1741692440.38 total 46.51 last 542.08) - open/close 30000 (time 1741692448.77 total 54.90 last 545.92) - open/close 35289 (time 1741692458.77 total 64.90 last 528.81) - open/close 40000 (time 1741692467.84 total 73.98 last 519.18) - open/close 44319 (time 1741692477.84 total 83.98 last 431.86) - open/close 49265 (time 1741692487.84 total 93.98 last 494.57) - open/close 54295 (time 1741692497.85 total 103.98 last 502.91) - open/close 59465 (time 1741692507.85 total 113.98 last 516.92) - open/close 64540 (time 1741692517.85 total 123.98 last 507.38) total: 64768 open/close in 124.65 seconds: 519.59 ops/second - /unlink 9387 (time 1741692529.40 total 10.00 last 938.69) - /unlink 18571 (time 1741692539.40 total 20.00 last 918.38) - /unlink 27832 (time 1741692549.40 total 30.00 last 926.08) - /unlink 30000 (time 1741692551.67 total 32.26 last 957.65) - /unlink 39302 (time 1741692561.67 total 42.26 last 930.17) - /unlink 48636 (time 1741692571.67 total 52.26 last 933.39) - /unlink 58009 (time 1741692581.67 total 62.27 last 937.22) - /unlink 60000 (time 1741692584.16 total 64.76 last 797.59) total: 64768 /unlink in 69.88 seconds: 926.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4896 (time 1741692629.51 total 10.00 last 489.55) - open/close 10000 (time 1741692639.43 total 19.92 last 514.66) - open/close 14595 (time 1741692649.43 total 29.92 last 459.39) - open/close 19268 (time 1741692659.43 total 39.92 last 467.23) - open/close 23432 (time 1741692669.43 total 49.92 last 416.32) - open/close 27910 (time 1741692679.43 total 59.93 last 447.74) - open/close 30000 (time 1741692684.34 total 64.83 last 426.00) - open/close 34134 (time 1741692694.34 total 74.83 last 413.36) - open/close 38098 (time 1741692704.34 total 84.83 last 396.32) - open/close 40000 (time 1741692709.26 total 89.76 last 386.45) - open/close 44127 (time 1741692719.27 total 99.76 last 412.63) - open/close 48318 (time 1741692729.27 total 109.76 last 419.09) - open/close 50000 (time 1741692732.80 total 113.30 last 475.23) - open/close 54731 (time 1741692742.81 total 123.30 last 473.05) - open/close 59822 (time 1741692752.81 total 133.30 last 509.08) total: 64768 open/close in 142.67 seconds: 453.96 ops/second - /unlink 9262 (time 1741692773.01 total 10.00 last 926.11) - /unlink 17986 (time 1741692783.01 total 20.00 last 872.33) - /unlink 20000 (time 1741692785.46 total 22.46 last 819.81) - /unlink 26408 (time 1741692795.46 total 32.46 last 640.78) - /unlink 30000 (time 1741692801.27 total 38.26 last 618.88) - /unlink 36737 (time 1741692811.27 total 48.26 last 673.63) - /unlink 40000 (time 1741692816.62 total 53.62 last 609.23) - /unlink 46896 (time 1741692826.62 total 63.62 last 689.59) - /unlink 50000 (time 1741692831.78 total 68.78 last 601.54) - /unlink 55980 (time 1741692841.79 total 78.78 last 597.96) - /unlink 60000 (time 1741692849.44 total 86.43 last 525.47) total: 64768 /unlink in 95.64 seconds: 677.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3489 (time 1741692884.58 total 10.00 last 348.83) - open/close 6327 (time 1741692894.58 total 20.00 last 283.79) - open/close 9553 (time 1741692904.58 total 30.01 last 322.52) - open/close 12942 (time 1741692914.59 total 40.01 last 338.85) - open/close 15955 (time 1741692924.59 total 50.01 last 301.28) - open/close 18867 (time 1741692934.59 total 60.01 last 291.15) - open/close 20000 (time 1741692938.33 total 63.75 last 303.10) - open/close 22905 (time 1741692948.33 total 73.75 last 290.49) - open/close 25696 (time 1741692958.33 total 83.75 last 279.10) - open/close 28422 (time 1741692968.33 total 93.75 last 272.54) - open/close 30000 (time 1741692974.09 total 99.51 last 273.94) - open/close 32644 (time 1741692984.09 total 109.51 last 264.40) - open/close 35339 (time 1741692994.09 total 119.51 last 269.47) - open/close 38430 (time 1741693004.09 total 129.51 last 309.09) - open/close 40000 (time 1741693008.85 total 134.27 last 330.12) - open/close 42719 (time 1741693018.85 total 144.27 last 271.90) - open/close 45295 (time 1741693028.85 total 154.27 last 257.55) - open/close 47953 (time 1741693038.85 total 164.27 last 265.71) - open/close 50000 (time 1741693046.94 total 172.37 last 252.96) - open/close 52645 (time 1741693056.94 total 182.37 last 264.49) - open/close 55657 (time 1741693066.95 total 192.37 last 301.15) - open/close 58421 (time 1741693076.95 total 202.37 last 276.37) - open/close 60000 (time 1741693083.04 total 208.46 last 259.07) - open/close 62521 (time 1741693093.04 total 218.46 last 252.08) total: 64768 open/close in 226.96 seconds: 285.37 ops/second - /unlink 4837 (time 1741693112.85 total 10.00 last 483.67) - /unlink 9796 (time 1741693122.85 total 20.00 last 495.87) - /unlink 16086 (time 1741693132.85 total 30.00 last 628.97) - /unlink 20000 (time 1741693137.45 total 34.60 last 851.71) - /unlink 25424 (time 1741693147.45 total 44.60 last 542.35) - /unlink 30000 (time 1741693156.37 total 53.52 last 513.07) - /unlink 34965 (time 1741693166.37 total 63.52 last 496.46) - /unlink 40000 (time 1741693176.11 total 73.26 last 516.77) - /unlink 45344 (time 1741693186.11 total 83.26 last 534.34) - /unlink 50000 (time 1741693195.19 total 92.34 last 512.71) - /unlink 54922 (time 1741693205.20 total 102.34 last 492.17) - /unlink 59772 (time 1741693215.20 total 112.35 last 484.94) total: 64768 /unlink in 121.03 seconds: 535.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 PASS 106 (1340s) Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg455-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1417 sec ========== 07:41:23 (1741693283) === conf-sanity: start cleanup 07:41:24 (1741693284) === === conf-sanity: finish cleanup 07:41:24 (1741693284) ===