-----============= acceptance-small: conf-sanity ============----- Wed May 21 02:34:12 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg432-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg432-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg432-server oleg432-server: oleg432-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg432-server: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg432-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:35:25 (1747809325) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) oleg432-server: oleg432-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5658 (time 1747809359.07 total 10.00 last 565.77) - open/close 10000 (time 1747809366.77 total 17.70 last 563.69) - open/close 15648 (time 1747809376.77 total 27.70 last 564.76) - open/close 20000 (time 1747809384.42 total 35.35 last 569.24) - open/close 25537 (time 1747809394.42 total 45.35 last 553.65) - open/close 30000 (time 1747809402.56 total 53.49 last 548.43) - open/close 35560 (time 1747809412.56 total 63.49 last 555.95) - open/close 40000 (time 1747809420.37 total 71.30 last 568.45) - open/close 45491 (time 1747809430.37 total 81.30 last 549.05) - open/close 50000 (time 1747809438.60 total 89.53 last 548.00) - open/close 55559 (time 1747809448.60 total 99.53 last 555.83) - open/close 60000 (time 1747809456.81 total 107.74 last 541.12) total: 64768 open/close in 116.48 seconds: 556.06 ops/second - /unlink 9640 (time 1747809476.36 total 10.00 last 963.93) - /unlink 18796 (time 1747809486.36 total 20.00 last 915.59) - /unlink 27985 (time 1747809496.36 total 30.00 last 918.85) - /unlink 30000 (time 1747809498.50 total 32.14 last 943.00) - /unlink 39082 (time 1747809508.50 total 42.14 last 908.16) - /unlink 48240 (time 1747809518.50 total 52.14 last 915.79) - /unlink 57509 (time 1747809528.50 total 62.14 last 926.84) - /unlink 60000 (time 1747809531.37 total 65.01 last 868.02) total: 64768 /unlink in 70.00 seconds: 925.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5291 (time 1747809572.40 total 10.00 last 529.04) - open/close 10000 (time 1747809581.48 total 19.08 last 518.75) - open/close 15368 (time 1747809591.48 total 29.08 last 536.78) - open/close 20000 (time 1747809600.21 total 37.81 last 530.42) - open/close 25229 (time 1747809610.21 total 47.81 last 522.90) - open/close 30000 (time 1747809619.14 total 56.74 last 534.51) - open/close 35321 (time 1747809629.14 total 66.74 last 532.06) - open/close 40000 (time 1747809638.13 total 75.74 last 519.97) - open/close 45105 (time 1747809648.14 total 85.74 last 510.41) - open/close 50000 (time 1747809657.56 total 95.16 last 519.36) - open/close 55153 (time 1747809667.56 total 105.16 last 515.27) - open/close 60000 (time 1747809677.10 total 114.70 last 508.25) total: 64768 open/close in 123.95 seconds: 522.51 ops/second - /unlink 9240 (time 1747809697.20 total 10.00 last 923.99) - /unlink 18141 (time 1747809707.20 total 20.00 last 890.10) - /unlink 27335 (time 1747809717.20 total 30.00 last 919.36) - /unlink 30000 (time 1747809719.97 total 32.77 last 962.39) - /unlink 38907 (time 1747809729.97 total 42.77 last 890.66) - /unlink 47946 (time 1747809739.97 total 52.77 last 903.81) - /unlink 50000 (time 1747809742.36 total 55.15 last 862.04) - /unlink 58886 (time 1747809752.36 total 65.15 last 888.59) total: 64768 /unlink in 71.96 seconds: 900.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4561 (time 1747809798.54 total 10.00 last 456.04) - open/close 8066 (time 1747809808.55 total 20.00 last 350.43) - open/close 10000 (time 1747809814.64 total 26.10 last 317.27) - open/close 13322 (time 1747809824.64 total 36.10 last 332.16) - open/close 16978 (time 1747809834.64 total 46.10 last 365.53) - open/close 20000 (time 1747809843.22 total 54.67 last 352.59) - open/close 23343 (time 1747809853.22 total 64.67 last 334.27) - open/close 26462 (time 1747809863.22 total 74.67 last 311.90) - open/close 29966 (time 1747809873.22 total 84.68 last 350.35) - open/close 34575 (time 1747809883.22 total 94.68 last 460.90) - open/close 38883 (time 1747809893.22 total 104.68 last 430.79) - open/close 40000 (time 1747809895.98 total 107.44 last 403.73) - open/close 43749 (time 1747809905.99 total 117.44 last 374.87) - open/close 47620 (time 1747809915.99 total 127.44 last 387.04) - open/close 50000 (time 1747809921.51 total 132.96 last 431.20) - open/close 53687 (time 1747809931.51 total 142.97 last 368.66) - open/close 57395 (time 1747809941.51 total 152.97 last 370.71) - open/close 60000 (time 1747809949.40 total 160.86 last 329.99) - open/close 63371 (time 1747809959.41 total 170.86 last 337.05) total: 64768 open/close in 174.06 seconds: 372.11 ops/second - /unlink 8370 (time 1747809973.58 total 10.00 last 836.93) - /unlink 16287 (time 1747809983.58 total 20.00 last 791.65) - /unlink 20000 (time 1747809987.63 total 24.05 last 917.34) - /unlink 27262 (time 1747809997.63 total 34.05 last 726.05) - /unlink 30000 (time 1747810001.36 total 37.79 last 732.63) - /unlink 38944 (time 1747810011.37 total 47.79 last 894.31) - /unlink 46568 (time 1747810021.37 total 57.79 last 762.35) - /unlink 50000 (time 1747810025.23 total 61.65 last 888.86) - /unlink 56132 (time 1747810035.23 total 71.65 last 613.17) - /unlink 60000 (time 1747810040.54 total 76.96 last 728.67) total: 64768 /unlink in 83.46 seconds: 776.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3430 (time 1747810073.51 total 10.00 last 342.94) - open/close 6394 (time 1747810083.51 total 20.00 last 296.38) - open/close 10000 (time 1747810092.16 total 28.65 last 416.80) - open/close 14046 (time 1747810102.16 total 38.66 last 404.54) - open/close 17318 (time 1747810112.16 total 48.66 last 327.13) - open/close 20000 (time 1747810120.09 total 56.58 last 338.41) - open/close 23831 (time 1747810130.09 total 66.59 last 383.00) - open/close 27355 (time 1747810140.10 total 76.59 last 352.30) - open/close 30000 (time 1747810147.16 total 83.66 last 374.14) - open/close 33752 (time 1747810157.17 total 93.66 last 375.10) - open/close 36533 (time 1747810167.17 total 103.66 last 278.01) - open/close 39740 (time 1747810177.17 total 113.67 last 320.62) - open/close 44124 (time 1747810187.17 total 123.67 last 438.38) - open/close 47996 (time 1747810197.18 total 133.67 last 387.12) - open/close 50000 (time 1747810202.29 total 138.78 last 392.08) - open/close 54324 (time 1747810212.29 total 148.78 last 432.32) - open/close 58168 (time 1747810222.29 total 158.78 last 384.38) - open/close 60000 (time 1747810227.99 total 164.48 last 321.30) total: 64768 open/close in 174.38 seconds: 371.41 ops/second - /unlink 8038 (time 1747810248.68 total 10.00 last 803.68) - /unlink 10000 (time 1747810251.66 total 12.98 last 658.15) - /unlink 17603 (time 1747810261.66 total 22.98 last 760.28) - /unlink 20000 (time 1747810265.72 total 27.04 last 590.94) - /unlink 25601 (time 1747810275.72 total 37.04 last 560.04) - /unlink 30000 (time 1747810284.46 total 45.78 last 503.55) - /unlink 36121 (time 1747810294.46 total 55.78 last 612.01) - /unlink 40000 (time 1747810300.21 total 61.52 last 675.05) - /unlink 49297 (time 1747810310.21 total 71.52 last 929.66) - /unlink 58909 (time 1747810320.21 total 81.52 last 961.13) total: 64768 /unlink in 87.53 seconds: 739.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4451 (time 1747810359.56 total 10.00 last 444.97) - open/close 8013 (time 1747810369.56 total 20.01 last 356.12) - open/close 10000 (time 1747810376.36 total 26.80 last 292.21) - open/close 14502 (time 1747810386.36 total 36.81 last 450.02) - open/close 18817 (time 1747810396.37 total 46.81 last 431.48) - open/close 20000 (time 1747810398.85 total 49.29 last 476.16) - open/close 23102 (time 1747810408.85 total 59.30 last 310.14) - open/close 26725 (time 1747810418.85 total 69.30 last 362.20) - open/close 30000 (time 1747810428.68 total 79.12 last 333.39) - open/close 33186 (time 1747810438.68 total 89.12 last 318.57) - open/close 37270 (time 1747810448.68 total 99.12 last 408.32) - open/close 40000 (time 1747810456.35 total 106.79 last 356.11) - open/close 44055 (time 1747810466.35 total 116.79 last 405.44) - open/close 47289 (time 1747810476.35 total 126.80 last 323.29) - open/close 50000 (time 1747810485.04 total 135.49 last 311.86) - open/close 52783 (time 1747810495.05 total 145.49 last 278.27) - open/close 55593 (time 1747810505.05 total 155.49 last 281.00) - open/close 58836 (time 1747810515.05 total 165.49 last 324.30) - open/close 60000 (time 1747810519.14 total 169.59 last 284.15) - open/close 63127 (time 1747810529.14 total 179.59 last 312.64) total: 64768 open/close in 185.60 seconds: 348.96 ops/second - /unlink 5208 (time 1747810546.33 total 10.00 last 520.77) - /unlink 10000 (time 1747810555.04 total 18.71 last 550.37) - /unlink 16189 (time 1747810565.04 total 28.71 last 618.86) - /unlink 20000 (time 1747810571.91 total 35.58 last 554.75) - /unlink 25430 (time 1747810581.91 total 45.58 last 542.94) - /unlink 30000 (time 1747810588.25 total 51.92 last 721.05) - /unlink 35355 (time 1747810598.25 total 61.92 last 535.46) - /unlink 40000 (time 1747810605.82 total 69.49 last 613.68) - /unlink 46228 (time 1747810615.82 total 79.49 last 622.74) - /unlink 50000 (time 1747810621.32 total 84.98 last 686.21) - /unlink 58493 (time 1747810631.32 total 94.98 last 849.28) total: 64768 /unlink in 102.24 seconds: 633.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 PASS 106 (1353s) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 oleg432-server: oleg432-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg432-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1444 sec ========== 02:58:17 (1747810697) === conf-sanity: start cleanup 02:58:17 (1747810697) === === conf-sanity: finish cleanup 02:58:18 (1747810698) ===