-----============= acceptance-small: conf-sanity ============----- Sun Feb 2 02:01:32 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg338-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg338-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg338-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg338-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg338-server oleg338-server: oleg338-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing load_modules_local oleg338-server: Loading modules from /home/green/git/lustre-release/lustre oleg338-server: detected 4 online CPUs by sysfs oleg338-server: MODOPTS_LIBCFS= oleg338-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg338-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg338-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg338-server: oleg338-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg338-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:02:31 (1738479751) Stopping clients: oleg338-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg338-client.virtnet /mnt/lustre2 (opts:-f) oleg338-server: oleg338-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing load_modules_local oleg338-server: Loading modules from /home/green/git/lustre-release/lustre oleg338-server: detected 4 online CPUs by sysfs oleg338-server: MODOPTS_LIBCFS= oleg338-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg338-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg338-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 6029 (time 1738479785.46 total 10.00 last 602.86) - open/close 10000 (time 1738479792.08 total 16.63 last 599.02) - open/close 15267 (time 1738479802.09 total 26.63 last 526.65) - open/close 20000 (time 1738479811.86 total 36.40 last 484.32) - open/close 25758 (time 1738479821.86 total 46.40 last 575.79) - open/close 30000 (time 1738479828.73 total 53.27 last 617.66) - open/close 36123 (time 1738479838.73 total 63.27 last 612.27) - open/close 40000 (time 1738479846.15 total 70.70 last 522.10) - open/close 45822 (time 1738479856.15 total 80.70 last 582.17) - open/close 50000 (time 1738479863.14 total 87.68 last 598.35) - open/close 55185 (time 1738479873.14 total 97.68 last 518.46) - open/close 60000 (time 1738479882.45 total 107.00 last 516.79) total: 64768 open/close in 116.18 seconds: 557.49 ops/second - /unlink 8941 (time 1738479902.44 total 10.00 last 894.07) - /unlink 17528 (time 1738479912.44 total 20.00 last 858.54) - /unlink 20000 (time 1738479915.24 total 22.79 last 885.71) - /unlink 28571 (time 1738479925.24 total 32.79 last 857.07) - /unlink 37146 (time 1738479935.24 total 42.79 last 857.48) - /unlink 40000 (time 1738479938.54 total 46.10 last 862.65) - /unlink 48610 (time 1738479948.54 total 56.10 last 860.96) - /unlink 57282 (time 1738479958.54 total 66.10 last 867.17) - /unlink 60000 (time 1738479961.80 total 69.36 last 835.29) total: 64768 /unlink in 74.65 seconds: 867.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4955 (time 1738479999.09 total 10.00 last 495.43) - open/close 9744 (time 1738480009.09 total 20.00 last 478.86) - open/close 14189 (time 1738480019.09 total 30.00 last 444.42) - open/close 19129 (time 1738480029.09 total 40.00 last 493.99) - open/close 24094 (time 1738480039.09 total 50.01 last 496.43) - open/close 29063 (time 1738480049.09 total 60.01 last 496.86) - open/close 34096 (time 1738480059.09 total 70.01 last 503.22) - open/close 39046 (time 1738480069.09 total 80.01 last 494.99) - open/close 44226 (time 1738480079.09 total 90.01 last 517.99) - open/close 49339 (time 1738480089.10 total 100.01 last 511.24) - open/close 54445 (time 1738480099.10 total 110.01 last 510.54) - open/close 59501 (time 1738480109.10 total 120.01 last 505.55) - open/close 64545 (time 1738480119.10 total 130.01 last 504.34) total: 64768 open/close in 130.53 seconds: 496.20 ops/second - /unlink 8607 (time 1738480130.48 total 10.00 last 860.69) - /unlink 17589 (time 1738480140.48 total 20.00 last 898.12) - /unlink 20000 (time 1738480143.39 total 22.91 last 829.95) - /unlink 28704 (time 1738480153.39 total 32.91 last 870.37) - /unlink 37217 (time 1738480163.39 total 42.91 last 851.22) - /unlink 40000 (time 1738480166.54 total 46.06 last 882.66) - /unlink 48747 (time 1738480176.54 total 56.06 last 874.67) - /unlink 57678 (time 1738480186.54 total 66.06 last 893.10) - /unlink 60000 (time 1738480189.17 total 68.68 last 885.14) total: 64768 /unlink in 73.88 seconds: 876.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5325 (time 1738480229.63 total 10.00 last 532.46) - open/close 10000 (time 1738480238.44 total 18.81 last 530.88) - open/close 14860 (time 1738480248.44 total 28.81 last 485.93) - open/close 20000 (time 1738480258.14 total 38.51 last 529.93) - open/close 25312 (time 1738480268.14 total 48.51 last 531.14) - open/close 30000 (time 1738480276.80 total 57.17 last 541.08) - open/close 35170 (time 1738480286.80 total 67.17 last 516.92) - open/close 40000 (time 1738480295.89 total 76.27 last 531.22) - open/close 45223 (time 1738480305.90 total 86.27 last 522.25) - open/close 50000 (time 1738480315.08 total 95.46 last 519.87) - open/close 55170 (time 1738480325.09 total 105.46 last 516.95) - open/close 60000 (time 1738480334.58 total 114.95 last 508.81) total: 64768 open/close in 123.96 seconds: 522.51 ops/second - /unlink 9005 (time 1738480354.40 total 10.00 last 900.45) - /unlink 17403 (time 1738480364.40 total 20.00 last 839.75) - /unlink 20000 (time 1738480367.29 total 22.88 last 900.62) - /unlink 28940 (time 1738480377.29 total 32.89 last 893.96) - /unlink 37738 (time 1738480387.29 total 42.89 last 879.75) - /unlink 40000 (time 1738480389.83 total 45.42 last 891.30) - /unlink 49019 (time 1738480399.83 total 55.42 last 901.86) - /unlink 58218 (time 1738480409.83 total 65.42 last 919.86) total: 64768 /unlink in 72.28 seconds: 896.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5267 (time 1738480454.71 total 10.00 last 526.66) - open/close 10000 (time 1738480464.06 total 19.36 last 505.79) - open/close 15221 (time 1738480474.07 total 29.36 last 522.02) - open/close 20000 (time 1738480483.49 total 38.79 last 507.01) - open/close 24611 (time 1738480493.49 total 48.79 last 461.01) - open/close 29143 (time 1738480503.50 total 58.79 last 453.14) - open/close 34292 (time 1738480513.50 total 68.79 last 514.83) - open/close 39155 (time 1738480523.50 total 78.79 last 486.23) - open/close 44169 (time 1738480533.50 total 88.79 last 501.28) - open/close 48010 (time 1738480543.50 total 98.80 last 384.03) - open/close 50000 (time 1738480548.41 total 103.70 last 405.41) - open/close 53798 (time 1738480558.41 total 113.70 last 379.79) - open/close 57685 (time 1738480568.48 total 123.78 last 385.96) - open/close 60000 (time 1738480574.99 total 130.28 last 355.67) - open/close 63925 (time 1738480584.99 total 140.29 last 392.46) total: 64768 open/close in 142.42 seconds: 454.77 ops/second - /unlink 6494 (time 1738480598.15 total 10.00 last 649.34) - /unlink 10000 (time 1738480603.97 total 15.82 last 602.90) - /unlink 15894 (time 1738480613.97 total 25.82 last 589.39) - /unlink 20000 (time 1738480621.50 total 33.35 last 545.29) - /unlink 27246 (time 1738480631.50 total 43.35 last 724.53) - /unlink 30000 (time 1738480634.99 total 46.84 last 788.12) - /unlink 38393 (time 1738480645.00 total 56.84 last 839.25) - /unlink 46992 (time 1738480655.00 total 66.84 last 859.88) - /unlink 50000 (time 1738480658.39 total 70.24 last 886.52) - /unlink 59122 (time 1738480668.39 total 80.24 last 912.13) total: 64768 /unlink in 86.32 seconds: 750.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3649 (time 1738480705.81 total 10.00 last 364.88) - open/close 8468 (time 1738480715.81 total 20.00 last 481.81) - open/close 10000 (time 1738480719.10 total 23.29 last 466.13) - open/close 13606 (time 1738480729.10 total 33.29 last 360.56) - open/close 18125 (time 1738480739.10 total 43.29 last 451.85) - open/close 20000 (time 1738480743.65 total 47.84 last 412.06) - open/close 24234 (time 1738480753.65 total 57.84 last 423.40) - open/close 28078 (time 1738480763.65 total 67.84 last 384.36) - open/close 30000 (time 1738480767.66 total 71.85 last 480.00) - open/close 34841 (time 1738480777.66 total 81.85 last 484.09) - open/close 39177 (time 1738480787.66 total 91.85 last 433.53) - open/close 43495 (time 1738480797.66 total 101.85 last 431.74) - open/close 48221 (time 1738480807.66 total 111.85 last 472.51) - open/close 50000 (time 1738480811.75 total 115.94 last 435.43) - open/close 53130 (time 1738480821.75 total 125.94 last 312.96) - open/close 55829 (time 1738480831.75 total 135.94 last 269.89) - open/close 58530 (time 1738480841.75 total 145.94 last 270.04) - open/close 60000 (time 1738480845.41 total 149.60 last 401.74) - open/close 63332 (time 1738480855.41 total 159.60 last 333.12) total: 64768 open/close in 163.60 seconds: 395.89 ops/second - /unlink 7539 (time 1738480870.25 total 10.00 last 753.90) - /unlink 10000 (time 1738480874.34 total 14.09 last 601.84) - /unlink 15823 (time 1738480884.34 total 24.09 last 582.20) - /unlink 20000 (time 1738480890.72 total 30.47 last 654.92) - /unlink 27039 (time 1738480900.72 total 40.47 last 703.80) - /unlink 30000 (time 1738480905.82 total 45.57 last 580.44) - /unlink 35087 (time 1738480915.82 total 55.57 last 508.61) - /unlink 40000 (time 1738480924.39 total 64.14 last 573.48) - /unlink 45394 (time 1738480934.39 total 74.14 last 539.38) - /unlink 50000 (time 1738480941.95 total 81.70 last 609.35) - /unlink 57894 (time 1738480951.95 total 91.70 last 789.38) - /unlink 60000 (time 1738480954.26 total 94.01 last 909.82) total: 64768 /unlink in 100.42 seconds: 644.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg338-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server unloading modules via unload_modules_local on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 PASS 106 (1250s) Stopping clients: oleg338-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg338-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 oleg338-server: oleg338-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing load_modules_local oleg338-server: Loading modules from /home/green/git/lustre-release/lustre oleg338-server: detected 4 online CPUs by sysfs oleg338-server: MODOPTS_LIBCFS= oleg338-server: Force libcfs to create 2 CPU partitions oleg338-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg338-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1327 sec ========== 02:23:40 (1738481020) === conf-sanity: start cleanup 02:23:41 (1738481021) === === conf-sanity: finish cleanup 02:23:41 (1738481021) ===