-----============= acceptance-small: conf-sanity ============----- Mon Mar 10 13:09:19 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg255-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 32b 32c 32e 32h 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg255-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg255-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:10:27 (1741626627) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2241 (time 1741626667.17 total 10.00 last 224.09) - open/close 4257 (time 1741626677.17 total 20.00 last 201.57) - open/close 6170 (time 1741626687.17 total 30.00 last 191.27) - open/close 7970 (time 1741626697.17 total 40.00 last 179.99) - open/close 10000 (time 1741626705.89 total 48.72 last 232.83) - open/close 12442 (time 1741626715.89 total 58.72 last 244.16) - open/close 15375 (time 1741626725.89 total 68.72 last 293.29) - open/close 18223 (time 1741626735.89 total 78.73 last 284.75) - open/close 20000 (time 1741626742.74 total 85.57 last 259.69) - open/close 22451 (time 1741626752.74 total 95.57 last 245.03) - open/close 25286 (time 1741626762.75 total 105.58 last 283.26) - open/close 28244 (time 1741626772.75 total 115.58 last 295.74) - open/close 30000 (time 1741626777.96 total 120.79 last 337.22) - open/close 33280 (time 1741626787.96 total 130.79 last 327.98) - open/close 36770 (time 1741626797.96 total 140.79 last 348.97) - open/close 40000 (time 1741626807.73 total 150.56 last 330.67) - open/close 43056 (time 1741626817.73 total 160.56 last 305.54) - open/close 46211 (time 1741626827.73 total 170.56 last 315.49) - open/close 49305 (time 1741626837.73 total 180.56 last 309.31) - open/close 50000 (time 1741626839.92 total 182.75 last 318.27) - open/close 53336 (time 1741626849.92 total 192.75 last 333.56) - open/close 56829 (time 1741626859.92 total 202.75 last 349.26) - open/close 59918 (time 1741626869.92 total 212.75 last 308.85) - open/close 63501 (time 1741626879.92 total 222.75 last 358.28) total: 64768 open/close in 226.34 seconds: 286.15 ops/second - /unlink 5939 (time 1741626894.41 total 10.00 last 593.85) - /unlink 10000 (time 1741626899.87 total 15.46 last 743.68) - /unlink 16078 (time 1741626909.87 total 25.46 last 607.77) - /unlink 20000 (time 1741626916.66 total 32.25 last 578.06) - /unlink 25611 (time 1741626926.66 total 42.25 last 561.09) - /unlink 30000 (time 1741626933.83 total 49.42 last 611.67) - /unlink 35774 (time 1741626943.84 total 59.42 last 577.34) - /unlink 40000 (time 1741626951.23 total 66.82 last 571.72) - /unlink 45166 (time 1741626961.23 total 76.82 last 516.57) - /unlink 49502 (time 1741626971.23 total 86.82 last 433.57) - /unlink 53433 (time 1741626981.23 total 96.82 last 393.10) - /unlink 57522 (time 1741626991.23 total 106.82 last 408.83) - /unlink 60000 (time 1741626996.69 total 112.27 last 454.15) total: 64768 /unlink in 121.16 seconds: 534.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2700 (time 1741627031.14 total 10.04 last 268.88) - open/close 5965 (time 1741627041.14 total 20.04 last 326.47) - open/close 8867 (time 1741627051.14 total 30.04 last 290.17) - open/close 10000 (time 1741627054.71 total 33.61 last 317.88) - open/close 12903 (time 1741627064.75 total 43.65 last 289.06) - open/close 16096 (time 1741627074.75 total 53.65 last 319.24) - open/close 19107 (time 1741627084.75 total 63.65 last 301.04) - open/close 20000 (time 1741627087.80 total 66.70 last 293.25) - open/close 22855 (time 1741627097.80 total 76.70 last 285.40) - open/close 25655 (time 1741627107.81 total 86.71 last 279.90) - open/close 28583 (time 1741627117.81 total 96.71 last 292.80) - open/close 30000 (time 1741627122.43 total 101.33 last 306.71) - open/close 32980 (time 1741627132.43 total 111.33 last 297.98) - open/close 35621 (time 1741627142.43 total 121.33 last 263.98) - open/close 38385 (time 1741627152.43 total 131.33 last 276.35) - open/close 40000 (time 1741627157.81 total 136.71 last 300.14) - open/close 43195 (time 1741627167.81 total 146.72 last 319.45) - open/close 46679 (time 1741627177.84 total 156.74 last 347.42) - open/close 50000 (time 1741627187.45 total 166.35 last 345.82) - open/close 53736 (time 1741627197.45 total 176.35 last 373.56) - open/close 57891 (time 1741627207.45 total 186.35 last 415.48) - open/close 60000 (time 1741627213.12 total 192.02 last 371.96) - open/close 64558 (time 1741627223.12 total 202.02 last 455.68) total: 64768 open/close in 202.58 seconds: 319.72 ops/second - /unlink 8172 (time 1741627234.53 total 10.00 last 817.18) - /unlink 10000 (time 1741627236.66 total 12.12 last 860.61) - /unlink 18427 (time 1741627246.66 total 22.13 last 842.63) - /unlink 20000 (time 1741627248.67 total 24.14 last 781.98) - /unlink 28806 (time 1741627258.67 total 34.14 last 880.58) - /unlink 37374 (time 1741627268.67 total 44.14 last 856.76) - /unlink 40000 (time 1741627271.99 total 47.45 last 792.14) - /unlink 48665 (time 1741627281.99 total 57.45 last 866.45) - /unlink 57378 (time 1741627291.99 total 67.45 last 871.24) - /unlink 60000 (time 1741627294.67 total 70.14 last 976.76) total: 64768 /unlink in 75.00 seconds: 863.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5106 (time 1741627335.48 total 10.00 last 510.55) - open/close 10000 (time 1741627344.14 total 18.66 last 565.14) - open/close 15088 (time 1741627354.14 total 28.66 last 508.73) - open/close 20000 (time 1741627362.85 total 37.37 last 564.20) - open/close 25396 (time 1741627372.85 total 47.37 last 539.59) - open/close 30000 (time 1741627381.08 total 55.60 last 559.45) - open/close 35442 (time 1741627391.08 total 65.60 last 544.11) - open/close 40000 (time 1741627400.69 total 75.21 last 474.12) - open/close 44810 (time 1741627410.69 total 85.21 last 480.99) - open/close 49307 (time 1741627420.69 total 95.22 last 449.61) - open/close 54317 (time 1741627430.69 total 105.22 last 500.99) - open/close 59351 (time 1741627440.70 total 115.22 last 503.30) - open/close 64459 (time 1741627450.70 total 125.22 last 510.79) total: 64768 open/close in 125.86 seconds: 514.61 ops/second - /unlink 8949 (time 1741627462.16 total 10.00 last 894.83) - /unlink 17423 (time 1741627472.16 total 20.00 last 847.33) - /unlink 20000 (time 1741627475.29 total 23.13 last 823.42) - /unlink 28637 (time 1741627485.29 total 33.13 last 863.64) - /unlink 37421 (time 1741627495.29 total 43.13 last 878.33) - /unlink 40000 (time 1741627498.19 total 46.04 last 887.94) - /unlink 48163 (time 1741627508.19 total 56.04 last 816.26) - /unlink 50000 (time 1741627510.30 total 58.14 last 873.23) - /unlink 58880 (time 1741627520.30 total 68.14 last 887.90) total: 64768 /unlink in 74.68 seconds: 867.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4539 (time 1741627561.11 total 10.00 last 453.84) - open/close 7658 (time 1741627571.11 total 20.00 last 311.85) - open/close 10000 (time 1741627579.89 total 28.79 last 266.68) - open/close 14383 (time 1741627589.89 total 38.79 last 438.30) - open/close 19758 (time 1741627599.89 total 48.79 last 537.50) - open/close 23895 (time 1741627609.89 total 58.79 last 413.67) - open/close 29327 (time 1741627619.90 total 68.79 last 543.13) - open/close 34756 (time 1741627629.90 total 78.79 last 542.89) - open/close 39273 (time 1741627639.90 total 88.79 last 451.67) - open/close 43748 (time 1741627649.90 total 98.79 last 447.47) - open/close 48150 (time 1741627659.90 total 108.79 last 440.19) - open/close 50000 (time 1741627664.27 total 113.16 last 423.52) - open/close 54366 (time 1741627674.27 total 123.16 last 436.59) - open/close 59338 (time 1741627684.27 total 133.16 last 497.13) - open/close 64610 (time 1741627694.27 total 143.16 last 527.18) total: 64768 open/close in 143.51 seconds: 451.31 ops/second - /unlink 9400 (time 1741627705.42 total 10.00 last 939.99) - /unlink 18640 (time 1741627715.42 total 20.00 last 923.96) - /unlink 27630 (time 1741627725.42 total 30.00 last 898.79) - /unlink 30000 (time 1741627728.17 total 32.75 last 862.02) - /unlink 37880 (time 1741627738.17 total 42.75 last 787.92) - /unlink 40000 (time 1741627740.97 total 45.55 last 757.49) - /unlink 48429 (time 1741627750.97 total 55.55 last 842.87) - /unlink 57143 (time 1741627760.97 total 65.55 last 871.31) - /unlink 60000 (time 1741627764.33 total 68.91 last 850.37) total: 64768 /unlink in 74.24 seconds: 872.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5042 (time 1741627806.34 total 10.00 last 504.14) - open/close 9855 (time 1741627816.34 total 20.00 last 481.25) - open/close 14994 (time 1741627826.34 total 30.00 last 513.82) - open/close 20000 (time 1741627836.03 total 39.69 last 516.85) - open/close 25043 (time 1741627846.03 total 49.69 last 504.26) - open/close 30000 (time 1741627855.95 total 59.61 last 499.54) - open/close 35104 (time 1741627865.95 total 69.61 last 510.39) - open/close 40000 (time 1741627875.37 total 79.03 last 520.13) - open/close 45088 (time 1741627885.37 total 89.03 last 508.78) - open/close 50000 (time 1741627895.01 total 98.67 last 509.27) - open/close 55046 (time 1741627905.01 total 108.67 last 504.52) - open/close 60000 (time 1741627914.96 total 118.62 last 497.85) - open/close 63784 (time 1741627924.97 total 128.63 last 378.35) total: 64768 open/close in 130.94 seconds: 494.63 ops/second - /unlink 9054 (time 1741627938.12 total 10.00 last 905.32) - /unlink 18177 (time 1741627948.12 total 20.00 last 912.13) - /unlink 27391 (time 1741627958.12 total 30.00 last 921.40) - /unlink 30000 (time 1741627961.01 total 32.90 last 901.00) - /unlink 39100 (time 1741627971.01 total 42.90 last 909.96) - /unlink 48111 (time 1741627981.01 total 52.90 last 901.07) - /unlink 50000 (time 1741627983.04 total 54.93 last 930.99) - /unlink 59392 (time 1741627993.04 total 64.93 last 939.19) total: 64768 /unlink in 70.68 seconds: 916.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 PASS 106 (1420s) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1507 sec ========== 13:34:26 (1741628066) === conf-sanity: start cleanup 13:34:27 (1741628067) === === conf-sanity: finish cleanup 13:34:27 (1741628067) ===