-----============= acceptance-small: conf-sanity ============----- Tue Feb 25 06:59:57 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg123-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 32b 32c 32e 32h 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg123-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg123-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg123-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg123-server oleg123-server: oleg123-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions oleg123-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg123-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg123-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg123-server: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg123-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:01:02 (1740484862) Stopping clients: oleg123-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg123-client.virtnet /mnt/lustre2 (opts:-f) oleg123-server: oleg123-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg123-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg123-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3384 (time 1740484901.89 total 10.00 last 338.39) - open/close 6136 (time 1740484911.89 total 20.00 last 275.20) - open/close 8752 (time 1740484921.89 total 30.00 last 261.53) - open/close 10000 (time 1740484926.69 total 34.80 last 260.04) - open/close 12318 (time 1740484936.69 total 44.80 last 231.76) - open/close 15207 (time 1740484946.70 total 54.81 last 288.85) - open/close 18491 (time 1740484956.70 total 64.81 last 328.38) - open/close 20000 (time 1740484961.12 total 69.23 last 340.93) - open/close 22719 (time 1740484971.12 total 79.23 last 271.89) - open/close 26051 (time 1740484981.13 total 89.24 last 333.13) - open/close 29195 (time 1740484991.13 total 99.24 last 314.31) - open/close 30000 (time 1740484993.37 total 101.48 last 359.13) - open/close 33100 (time 1740485003.37 total 111.48 last 309.97) - open/close 36346 (time 1740485013.37 total 121.48 last 324.49) - open/close 38600 (time 1740485023.38 total 131.49 last 225.38) - open/close 40000 (time 1740485028.93 total 137.04 last 252.10) - open/close 42493 (time 1740485038.93 total 147.04 last 249.21) - open/close 44393 (time 1740485048.93 total 157.04 last 189.95) - open/close 46508 (time 1740485058.94 total 167.05 last 211.46) - open/close 48312 (time 1740485068.94 total 177.05 last 180.33) - open/close 50000 (time 1740485077.90 total 186.01 last 188.34) - open/close 51958 (time 1740485087.91 total 196.02 last 195.73) - open/close 53689 (time 1740485097.91 total 206.02 last 173.06) - open/close 56133 (time 1740485107.91 total 216.02 last 244.36) - open/close 58627 (time 1740485117.91 total 226.02 last 249.39) - open/close 60000 (time 1740485123.57 total 231.68 last 242.48) - open/close 62775 (time 1740485133.57 total 241.68 last 277.46) total: 64768 open/close in 246.96 seconds: 262.26 ops/second - /unlink 5871 (time 1740485149.88 total 10.00 last 587.03) - /unlink 10000 (time 1740485156.65 total 16.77 last 610.09) - /unlink 15050 (time 1740485166.66 total 26.77 last 504.85) - /unlink 20000 (time 1740485175.88 total 36.00 last 536.46) - /unlink 27249 (time 1740485185.88 total 46.00 last 724.89) - /unlink 30000 (time 1740485189.54 total 49.65 last 752.72) - /unlink 36054 (time 1740485199.54 total 59.66 last 605.24) - /unlink 40000 (time 1740485206.72 total 66.83 last 549.87) - /unlink 44734 (time 1740485216.72 total 76.83 last 473.35) - /unlink 49601 (time 1740485226.72 total 86.83 last 486.67) - /unlink 55136 (time 1740485236.72 total 96.83 last 553.48) - /unlink 60000 (time 1740485244.10 total 104.22 last 659.05) total: 64768 /unlink in 111.05 seconds: 583.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3361 (time 1740485279.31 total 10.00 last 336.07) - open/close 6645 (time 1740485289.31 total 20.00 last 328.33) - open/close 9994 (time 1740485299.31 total 30.00 last 334.88) - open/close 13755 (time 1740485309.32 total 40.00 last 376.06) - open/close 17940 (time 1740485319.32 total 50.01 last 418.47) - open/close 20000 (time 1740485324.65 total 55.34 last 386.33) - open/close 23510 (time 1740485334.65 total 65.34 last 350.94) - open/close 26541 (time 1740485344.65 total 75.34 last 303.04) - open/close 29953 (time 1740485354.65 total 85.34 last 341.11) - open/close 33172 (time 1740485364.66 total 95.34 last 321.88) - open/close 36889 (time 1740485374.66 total 105.35 last 371.64) - open/close 40000 (time 1740485382.10 total 112.79 last 417.83) - open/close 43865 (time 1740485392.10 total 122.79 last 386.44) - open/close 47669 (time 1740485402.11 total 132.79 last 380.33) - open/close 50000 (time 1740485408.37 total 139.06 last 371.96) - open/close 53655 (time 1740485418.37 total 149.06 last 365.45) - open/close 57828 (time 1740485428.38 total 159.07 last 417.21) - open/close 60000 (time 1740485433.93 total 164.62 last 390.81) - open/close 64225 (time 1740485443.93 total 174.62 last 422.48) total: 64768 open/close in 175.87 seconds: 368.27 ops/second - /unlink 6818 (time 1740485456.16 total 10.00 last 681.79) - /unlink 10000 (time 1740485460.36 total 14.20 last 756.77) - /unlink 17561 (time 1740485470.36 total 24.21 last 756.08) - /unlink 20000 (time 1740485473.35 total 27.20 last 815.68) - /unlink 27579 (time 1740485483.35 total 37.20 last 757.86) - /unlink 30000 (time 1740485486.71 total 40.55 last 721.23) - /unlink 38081 (time 1740485496.71 total 50.55 last 808.09) - /unlink 40000 (time 1740485499.17 total 53.01 last 779.91) - /unlink 48532 (time 1740485509.17 total 63.01 last 853.19) - /unlink 57048 (time 1740485519.17 total 73.01 last 851.59) - /unlink 60000 (time 1740485522.60 total 76.44 last 862.51) total: 64768 /unlink in 81.90 seconds: 790.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5056 (time 1740485560.27 total 10.00 last 505.60) - open/close 10000 (time 1740485569.96 total 19.70 last 509.91) - open/close 15228 (time 1740485579.96 total 29.70 last 522.71) - open/close 20000 (time 1740485589.34 total 39.07 last 509.07) - open/close 24904 (time 1740485599.34 total 49.07 last 490.36) - open/close 29831 (time 1740485609.34 total 59.07 last 492.70) - open/close 35061 (time 1740485619.34 total 69.07 last 522.91) - open/close 40000 (time 1740485629.08 total 78.82 last 507.04) - open/close 44774 (time 1740485639.08 total 88.82 last 477.39) - open/close 49591 (time 1740485649.08 total 98.82 last 481.66) - open/close 54038 (time 1740485659.08 total 108.82 last 444.69) - open/close 58443 (time 1740485669.08 total 118.82 last 440.45) - open/close 60000 (time 1740485672.31 total 122.04 last 482.44) total: 64768 open/close in 131.55 seconds: 492.33 ops/second - /unlink 8688 (time 1740485692.72 total 10.00 last 868.68) - /unlink 17457 (time 1740485702.72 total 20.00 last 876.88) - /unlink 20000 (time 1740485705.70 total 22.98 last 852.49) - /unlink 28325 (time 1740485715.70 total 32.98 last 832.49) - /unlink 36977 (time 1740485725.70 total 42.98 last 865.20) - /unlink 40000 (time 1740485729.14 total 46.43 last 878.58) - /unlink 48536 (time 1740485739.14 total 56.43 last 853.56) - /unlink 57268 (time 1740485749.14 total 66.43 last 873.12) - /unlink 60000 (time 1740485752.43 total 69.71 last 831.71) total: 64768 /unlink in 75.14 seconds: 861.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5084 (time 1740485791.02 total 10.00 last 508.40) - open/close 9926 (time 1740485801.02 total 20.00 last 484.16) - open/close 14934 (time 1740485811.02 total 30.00 last 500.78) - open/close 19489 (time 1740485821.02 total 40.00 last 455.44) - open/close 24252 (time 1740485831.02 total 50.00 last 476.29) - open/close 29047 (time 1740485841.02 total 60.00 last 479.47) - open/close 30000 (time 1740485843.29 total 62.27 last 421.06) - open/close 34502 (time 1740485853.29 total 72.27 last 450.16) - open/close 39085 (time 1740485863.29 total 82.27 last 458.25) - open/close 43691 (time 1740485873.29 total 92.27 last 460.60) - open/close 48364 (time 1740485883.29 total 102.27 last 467.29) - open/close 50000 (time 1740485887.01 total 105.99 last 439.74) - open/close 54556 (time 1740485897.01 total 115.99 last 455.55) - open/close 59208 (time 1740485907.01 total 125.99 last 465.15) - open/close 64118 (time 1740485917.01 total 135.99 last 490.96) total: 64768 open/close in 137.39 seconds: 471.41 ops/second - /unlink 8569 (time 1740485929.29 total 10.00 last 856.84) - /unlink 17719 (time 1740485939.29 total 20.00 last 914.92) - /unlink 20000 (time 1740485941.87 total 22.58 last 882.99) - /unlink 28810 (time 1740485951.87 total 32.59 last 880.95) - /unlink 37582 (time 1740485961.87 total 42.59 last 877.17) - /unlink 40000 (time 1740485964.79 total 45.51 last 828.05) - /unlink 48656 (time 1740485974.79 total 55.51 last 865.59) - /unlink 57510 (time 1740485984.80 total 65.51 last 885.33) - /unlink 60000 (time 1740485987.46 total 68.17 last 936.20) total: 64768 /unlink in 73.59 seconds: 880.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5557 (time 1740486030.88 total 10.00 last 555.70) - open/close 10000 (time 1740486039.00 total 18.13 last 546.83) - open/close 15406 (time 1740486049.00 total 28.13 last 540.60) - open/close 20000 (time 1740486057.61 total 36.73 last 533.79) - open/close 25449 (time 1740486067.61 total 46.73 last 544.85) - open/close 30000 (time 1740486076.27 total 55.40 last 525.28) - open/close 35406 (time 1740486086.27 total 65.40 last 540.54) - open/close 40000 (time 1740486094.98 total 74.10 last 527.70) - open/close 45197 (time 1740486104.98 total 84.10 last 519.61) - open/close 50000 (time 1740486114.43 total 93.55 last 508.48) - open/close 55192 (time 1740486124.43 total 103.55 last 519.15) - open/close 60000 (time 1740486133.76 total 112.88 last 515.49) - open/close 64427 (time 1740486143.76 total 122.88 last 442.70) total: 64768 open/close in 123.59 seconds: 524.05 ops/second - /unlink 9233 (time 1740486155.29 total 10.00 last 923.26) - /unlink 18087 (time 1740486165.29 total 20.00 last 885.38) - /unlink 20000 (time 1740486167.41 total 22.12 last 900.91) - /unlink 29262 (time 1740486177.41 total 32.12 last 926.15) - /unlink 38010 (time 1740486187.41 total 42.13 last 874.77) - /unlink 40000 (time 1740486189.57 total 44.28 last 921.37) - /unlink 49027 (time 1740486199.57 total 54.29 last 902.68) - /unlink 57743 (time 1740486209.57 total 64.29 last 871.49) - /unlink 60000 (time 1740486212.02 total 66.74 last 920.99) total: 64768 /unlink in 71.89 seconds: 900.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 PASS 106 (1403s) Stopping clients: oleg123-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg123-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 oleg123-server: oleg123-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions oleg123-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg123-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1484 sec ========== 07:24:42 (1740486282) === conf-sanity: start cleanup 07:24:42 (1740486282) === === conf-sanity: finish cleanup 07:24:43 (1740486283) ===