-----============= acceptance-small: conf-sanity ============----- Mon Apr 7 05:56:12 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg121-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg121-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg121-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg121-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg121-server oleg121-server: oleg121-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg121-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg121-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg121-server: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg121-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:57:02 (1744019822) Stopping clients: oleg121-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg121-client.virtnet /mnt/lustre2 (opts:-f) oleg121-server: oleg121-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg121-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg121-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4374 (time 1744019857.42 total 10.00 last 437.33) - open/close 8886 (time 1744019867.42 total 20.00 last 451.19) - open/close 10000 (time 1744019869.99 total 22.57 last 432.96) - open/close 14517 (time 1744019879.99 total 32.58 last 451.67) - open/close 18729 (time 1744019889.99 total 42.58 last 421.17) - open/close 20000 (time 1744019892.74 total 45.33 last 461.57) - open/close 24899 (time 1744019902.75 total 55.33 last 489.83) - open/close 30000 (time 1744019911.60 total 64.18 last 576.30) - open/close 35838 (time 1744019921.60 total 74.18 last 583.80) - open/close 40000 (time 1744019929.54 total 82.13 last 523.98) - open/close 45364 (time 1744019939.54 total 92.13 last 536.32) - open/close 50000 (time 1744019948.31 total 100.89 last 528.94) - open/close 55389 (time 1744019958.31 total 110.89 last 538.88) - open/close 60000 (time 1744019966.63 total 119.22 last 553.83) total: 64768 open/close in 128.16 seconds: 505.39 ops/second - /unlink 8848 (time 1744019986.36 total 10.00 last 884.68) - /unlink 17728 (time 1744019996.36 total 20.00 last 887.92) - /unlink 20000 (time 1744019999.02 total 22.66 last 853.96) - /unlink 28304 (time 1744020009.02 total 32.66 last 830.33) - /unlink 30000 (time 1744020011.09 total 34.73 last 820.10) - /unlink 38129 (time 1744020021.09 total 44.73 last 812.88) - /unlink 40000 (time 1744020023.30 total 46.94 last 847.78) - /unlink 48550 (time 1744020033.30 total 56.94 last 854.92) - /unlink 55886 (time 1744020043.30 total 66.94 last 733.55) - /unlink 60000 (time 1744020048.24 total 71.88 last 832.86) total: 64768 /unlink in 78.00 seconds: 830.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4365 (time 1744020085.71 total 10.00 last 436.47) - open/close 7800 (time 1744020095.71 total 20.00 last 343.49) - open/close 10000 (time 1744020101.33 total 25.62 last 391.43) - open/close 13712 (time 1744020111.34 total 35.62 last 371.15) - open/close 18124 (time 1744020121.34 total 45.62 last 441.16) - open/close 20000 (time 1744020125.47 total 49.76 last 453.79) - open/close 24712 (time 1744020135.47 total 59.76 last 471.18) - open/close 29104 (time 1744020145.47 total 69.76 last 439.13) - open/close 33854 (time 1744020155.47 total 79.76 last 474.92) - open/close 38725 (time 1744020165.48 total 89.76 last 487.06) - open/close 40000 (time 1744020168.12 total 92.41 last 481.80) - open/close 44998 (time 1744020178.12 total 102.41 last 499.76) - open/close 49956 (time 1744020188.12 total 112.41 last 495.80) - open/close 54901 (time 1744020198.12 total 122.41 last 494.47) - open/close 59475 (time 1744020208.12 total 132.41 last 457.33) - open/close 63972 (time 1744020218.13 total 142.41 last 449.66) total: 64768 open/close in 144.18 seconds: 449.22 ops/second - /unlink 8102 (time 1744020230.76 total 10.00 last 810.16) - /unlink 10000 (time 1744020232.95 total 12.19 last 866.24) - /unlink 17770 (time 1744020242.95 total 22.19 last 776.94) - /unlink 20000 (time 1744020245.79 total 25.03 last 785.14) - /unlink 28177 (time 1744020255.79 total 35.03 last 817.69) - /unlink 30000 (time 1744020257.86 total 37.10 last 881.88) - /unlink 38761 (time 1744020267.86 total 47.10 last 876.10) - /unlink 47411 (time 1744020277.86 total 57.10 last 864.93) - /unlink 50000 (time 1744020280.88 total 60.12 last 856.16) - /unlink 58624 (time 1744020290.88 total 70.12 last 862.39) total: 64768 /unlink in 76.75 seconds: 843.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5559 (time 1744020331.05 total 10.00 last 555.87) - open/close 10000 (time 1744020339.09 total 18.04 last 552.16) - open/close 15516 (time 1744020349.09 total 28.04 last 551.55) - open/close 20000 (time 1744020357.66 total 36.61 last 523.33) - open/close 25352 (time 1744020367.66 total 46.61 last 535.12) - open/close 30000 (time 1744020376.47 total 55.42 last 527.64) - open/close 35391 (time 1744020386.47 total 65.42 last 539.05) - open/close 40000 (time 1744020395.22 total 74.17 last 526.83) - open/close 44987 (time 1744020405.22 total 84.17 last 498.65) - open/close 50000 (time 1744020414.84 total 93.79 last 521.07) - open/close 55361 (time 1744020424.84 total 103.79 last 536.08) - open/close 60000 (time 1744020433.70 total 112.66 last 523.51) total: 64768 open/close in 121.49 seconds: 533.09 ops/second - /unlink 9229 (time 1744020453.36 total 10.00 last 922.77) - /unlink 18421 (time 1744020463.36 total 20.00 last 919.17) - /unlink 27500 (time 1744020473.36 total 30.00 last 907.83) - /unlink 30000 (time 1744020476.13 total 32.78 last 900.70) - /unlink 38416 (time 1744020486.13 total 42.78 last 841.59) - /unlink 47577 (time 1744020496.13 total 52.78 last 916.04) - /unlink 50000 (time 1744020498.65 total 55.30 last 961.27) - /unlink 59197 (time 1744020508.65 total 65.30 last 919.69) total: 64768 /unlink in 71.13 seconds: 910.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5164 (time 1744020551.64 total 10.00 last 516.36) - open/close 10000 (time 1744020560.84 total 19.20 last 525.68) - open/close 15288 (time 1744020570.84 total 29.20 last 528.74) - open/close 20000 (time 1744020579.87 total 38.23 last 521.69) - open/close 25308 (time 1744020589.87 total 48.24 last 530.72) - open/close 30000 (time 1744020598.79 total 57.16 last 525.93) - open/close 34600 (time 1744020608.80 total 67.16 last 459.92) - open/close 39195 (time 1744020618.80 total 77.16 last 459.46) - open/close 44281 (time 1744020628.80 total 87.16 last 508.52) - open/close 48808 (time 1744020638.80 total 97.16 last 452.65) - open/close 50000 (time 1744020641.41 total 99.77 last 457.07) - open/close 54969 (time 1744020651.41 total 109.77 last 496.80) - open/close 59642 (time 1744020661.41 total 119.77 last 467.30) - open/close 64479 (time 1744020671.41 total 129.77 last 483.61) total: 64768 open/close in 130.40 seconds: 496.69 ops/second - /unlink 9106 (time 1744020682.85 total 10.00 last 910.52) - /unlink 17844 (time 1744020692.85 total 20.00 last 873.71) - /unlink 20000 (time 1744020695.31 total 22.46 last 877.71) - /unlink 28622 (time 1744020705.31 total 32.46 last 862.19) - /unlink 36779 (time 1744020715.31 total 42.46 last 815.61) - /unlink 40000 (time 1744020718.72 total 45.87 last 944.52) - /unlink 47562 (time 1744020728.72 total 55.87 last 756.19) - /unlink 50000 (time 1744020732.23 total 59.38 last 694.85) - /unlink 58990 (time 1744020742.23 total 69.38 last 898.99) total: 64768 /unlink in 75.61 seconds: 856.60 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3790 (time 1744020781.99 total 10.00 last 378.99) - open/close 7326 (time 1744020791.99 total 20.00 last 353.57) - open/close 10000 (time 1744020799.72 total 27.73 last 345.81) - open/close 14070 (time 1744020809.72 total 37.73 last 406.95) - open/close 18724 (time 1744020819.72 total 47.74 last 465.35) - open/close 20000 (time 1744020822.42 total 50.43 last 473.54) - open/close 24751 (time 1744020832.42 total 60.43 last 475.03) - open/close 28807 (time 1744020842.42 total 70.43 last 405.50) - open/close 30000 (time 1744020846.14 total 74.15 last 321.00) - open/close 34488 (time 1744020856.14 total 84.15 last 448.69) - open/close 37813 (time 1744020866.14 total 94.15 last 332.48) - open/close 40000 (time 1744020871.42 total 99.43 last 414.50) - open/close 43823 (time 1744020881.42 total 109.43 last 382.27) - open/close 47804 (time 1744020891.44 total 119.45 last 397.27) - open/close 50000 (time 1744020895.89 total 123.90 last 493.68) - open/close 54551 (time 1744020905.89 total 133.90 last 455.06) - open/close 59086 (time 1744020915.89 total 143.90 last 453.45) - open/close 63992 (time 1744020925.89 total 153.90 last 490.57) total: 64768 open/close in 155.94 seconds: 415.34 ops/second - /unlink 7087 (time 1744020939.11 total 10.00 last 708.58) - /unlink 10000 (time 1744020942.32 total 13.22 last 905.34) - /unlink 17703 (time 1744020952.33 total 23.22 last 770.17) - /unlink 20000 (time 1744020956.39 total 27.29 last 565.10) - /unlink 25501 (time 1744020966.39 total 37.29 last 550.07) - /unlink 30000 (time 1744020972.86 total 43.76 last 695.43) - /unlink 36184 (time 1744020982.86 total 53.76 last 618.32) - /unlink 40000 (time 1744020989.17 total 60.06 last 605.12) - /unlink 47118 (time 1744020999.17 total 70.06 last 711.78) - /unlink 50000 (time 1744021002.81 total 73.70 last 792.42) - /unlink 58063 (time 1744021012.81 total 83.70 last 806.18) - /unlink 60000 (time 1744021015.92 total 86.81 last 622.50) total: 64768 /unlink in 93.29 seconds: 694.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 PASS 106 (1239s) Stopping clients: oleg121-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg121-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 oleg121-server: oleg121-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1308 sec ========== 06:18:00 (1744021080) === conf-sanity: start cleanup 06:18:01 (1744021081) === === conf-sanity: finish cleanup 06:18:01 (1744021081) ===