-----============= acceptance-small: conf-sanity ============----- Tue Jun 10 03:32:44 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg424-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg424-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg424-server oleg424-server: oleg424-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg424-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:34:15 (1749540855) Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) oleg424-server: oleg424-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg424-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3703 (time 1749540897.95 total 10.00 last 370.25) - open/close 7541 (time 1749540907.95 total 20.00 last 383.74) - open/close 10000 (time 1749540913.16 total 25.21 last 471.82) - open/close 14284 (time 1749540923.16 total 35.22 last 428.34) - open/close 19356 (time 1749540933.16 total 45.22 last 507.17) - open/close 24516 (time 1749540943.16 total 55.22 last 515.98) - open/close 29485 (time 1749540953.16 total 65.22 last 496.86) - open/close 33063 (time 1749540963.17 total 75.22 last 357.74) - open/close 36983 (time 1749540973.17 total 85.22 last 392.00) - open/close 40000 (time 1749540981.76 total 93.81 last 351.23) - open/close 43841 (time 1749540991.76 total 103.81 last 384.06) - open/close 47450 (time 1749541001.76 total 113.81 last 360.90) - open/close 50000 (time 1749541008.72 total 120.77 last 366.43) - open/close 53134 (time 1749541018.72 total 130.77 last 313.30) - open/close 56899 (time 1749541028.72 total 140.77 last 376.49) - open/close 60000 (time 1749541038.00 total 150.06 last 334.01) - open/close 63593 (time 1749541048.01 total 160.06 last 359.23) total: 64768 open/close in 163.20 seconds: 396.85 ops/second - /unlink 6459 (time 1749541062.06 total 10.00 last 645.77) - /unlink 10000 (time 1749541068.20 total 16.13 last 577.41) - /unlink 15752 (time 1749541078.20 total 26.14 last 575.18) - /unlink 20000 (time 1749541085.65 total 33.59 last 569.78) - /unlink 26103 (time 1749541095.65 total 43.59 last 610.24) - /unlink 30000 (time 1749541101.85 total 49.79 last 628.53) - /unlink 36500 (time 1749541111.85 total 59.79 last 650.00) - /unlink 40000 (time 1749541117.50 total 65.44 last 620.05) - /unlink 46416 (time 1749541127.50 total 75.44 last 641.55) - /unlink 50000 (time 1749541133.20 total 81.13 last 629.03) - /unlink 56613 (time 1749541143.20 total 91.14 last 661.24) - /unlink 60000 (time 1749541149.24 total 97.18 last 560.60) total: 64768 /unlink in 103.06 seconds: 628.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3107 (time 1749541186.21 total 10.00 last 310.66) - open/close 5812 (time 1749541196.22 total 20.01 last 270.38) - open/close 8487 (time 1749541206.22 total 30.01 last 267.49) - open/close 10000 (time 1749541211.64 total 35.43 last 279.16) - open/close 12901 (time 1749541221.64 total 45.43 last 289.98) - open/close 16641 (time 1749541231.64 total 55.43 last 373.97) - open/close 20000 (time 1749541240.97 total 64.76 last 359.88) - open/close 23405 (time 1749541250.98 total 74.77 last 340.44) - open/close 26645 (time 1749541260.98 total 84.77 last 323.99) - open/close 29961 (time 1749541270.98 total 94.77 last 331.51) - open/close 33087 (time 1749541280.98 total 104.77 last 312.49) - open/close 36108 (time 1749541290.98 total 114.77 last 302.08) - open/close 39594 (time 1749541300.99 total 124.78 last 348.50) - open/close 42628 (time 1749541310.99 total 134.78 last 303.34) - open/close 45728 (time 1749541320.99 total 144.78 last 309.93) - open/close 48709 (time 1749541330.99 total 154.78 last 298.09) - open/close 50000 (time 1749541335.75 total 159.54 last 271.12) - open/close 53175 (time 1749541345.75 total 169.55 last 317.42) - open/close 56463 (time 1749541355.76 total 179.55 last 328.71) - open/close 59394 (time 1749541365.76 total 189.55 last 293.09) - open/close 62805 (time 1749541375.76 total 199.55 last 341.09) total: 64768 open/close in 204.75 seconds: 316.32 ops/second - /unlink 6829 (time 1749541391.85 total 10.00 last 682.86) - /unlink 10000 (time 1749541396.99 total 15.15 last 616.37) - /unlink 16046 (time 1749541406.99 total 25.15 last 604.57) - /unlink 20000 (time 1749541413.34 total 31.49 last 623.48) - /unlink 26351 (time 1749541423.34 total 41.49 last 635.08) - /unlink 30000 (time 1749541429.08 total 47.23 last 635.53) - /unlink 36547 (time 1749541439.08 total 57.23 last 654.65) - /unlink 40000 (time 1749541444.73 total 62.89 last 610.57) - /unlink 46591 (time 1749541454.73 total 72.89 last 659.06) - /unlink 50000 (time 1749541459.79 total 77.95 last 673.72) - /unlink 58414 (time 1749541469.80 total 87.95 last 841.37) - /unlink 60000 (time 1749541471.80 total 89.95 last 790.47) total: 64768 /unlink in 95.06 seconds: 681.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5064 (time 1749541506.87 total 10.00 last 506.34) - open/close 10000 (time 1749541515.96 total 19.09 last 542.97) - open/close 15526 (time 1749541525.96 total 29.09 last 552.60) - open/close 20000 (time 1749541534.28 total 37.41 last 538.08) - open/close 24908 (time 1749541544.28 total 47.41 last 490.77) - open/close 29980 (time 1749541554.28 total 57.41 last 507.14) - open/close 35376 (time 1749541564.28 total 67.41 last 539.59) - open/close 40000 (time 1749541573.66 total 76.80 last 492.59) - open/close 44967 (time 1749541583.67 total 86.80 last 496.62) - open/close 50000 (time 1749541593.49 total 96.62 last 512.34) - open/close 55148 (time 1749541603.49 total 106.62 last 514.72) - open/close 60000 (time 1749541612.89 total 116.02 last 516.46) total: 64768 open/close in 125.33 seconds: 516.78 ops/second - /unlink 8941 (time 1749541633.08 total 10.00 last 894.07) - /unlink 18147 (time 1749541643.08 total 20.00 last 920.52) - /unlink 20000 (time 1749541645.31 total 22.23 last 831.32) - /unlink 28802 (time 1749541655.31 total 32.23 last 880.10) - /unlink 37576 (time 1749541665.31 total 42.23 last 877.37) - /unlink 40000 (time 1749541667.90 total 44.82 last 936.65) - /unlink 48922 (time 1749541677.90 total 54.82 last 892.19) - /unlink 57802 (time 1749541687.90 total 64.82 last 887.99) - /unlink 60000 (time 1749541690.37 total 67.29 last 890.05) total: 64768 /unlink in 72.51 seconds: 893.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5424 (time 1749541732.00 total 10.00 last 542.33) - open/close 10000 (time 1749541740.73 total 18.74 last 523.90) - open/close 15244 (time 1749541750.73 total 28.74 last 524.39) - open/close 20000 (time 1749541760.45 total 38.45 last 489.50) - open/close 23934 (time 1749541770.45 total 48.45 last 393.39) - open/close 28661 (time 1749541780.45 total 58.45 last 472.62) - open/close 30000 (time 1749541783.62 total 61.63 last 422.00) - open/close 34714 (time 1749541793.63 total 71.63 last 471.37) - open/close 39522 (time 1749541803.63 total 81.63 last 480.75) - open/close 44254 (time 1749541813.63 total 91.63 last 473.18) - open/close 49175 (time 1749541823.63 total 101.63 last 492.10) - open/close 54398 (time 1749541833.63 total 111.63 last 522.22) - open/close 59695 (time 1749541843.68 total 121.68 last 527.10) total: 64768 open/close in 131.26 seconds: 493.43 ops/second - /unlink 9032 (time 1749541864.30 total 10.00 last 903.13) - /unlink 17737 (time 1749541874.30 total 20.00 last 870.42) - /unlink 20000 (time 1749541877.26 total 22.97 last 762.67) - /unlink 29051 (time 1749541887.27 total 32.97 last 905.02) - /unlink 38320 (time 1749541897.27 total 42.97 last 926.90) - /unlink 47081 (time 1749541907.27 total 52.97 last 875.97) - /unlink 50000 (time 1749541911.09 total 56.79 last 763.42) - /unlink 57375 (time 1749541921.09 total 66.80 last 737.40) - /unlink 60000 (time 1749541925.10 total 70.81 last 654.42) total: 64768 /unlink in 76.02 seconds: 852.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5685 (time 1749541962.26 total 10.00 last 568.41) - open/close 10000 (time 1749541969.99 total 17.73 last 558.46) - open/close 14741 (time 1749541979.99 total 27.73 last 474.04) - open/close 19016 (time 1749541989.99 total 37.73 last 427.45) - open/close 24418 (time 1749541999.99 total 47.73 last 540.13) - open/close 29097 (time 1749542009.99 total 57.73 last 467.88) - open/close 30000 (time 1749542012.84 total 60.58 last 317.23) - open/close 33657 (time 1749542022.84 total 70.58 last 365.68) - open/close 37141 (time 1749542032.84 total 80.58 last 348.36) - open/close 40000 (time 1749542039.20 total 86.94 last 449.32) - open/close 43598 (time 1749542049.20 total 96.94 last 359.78) - open/close 47696 (time 1749542059.20 total 106.94 last 409.75) - open/close 50000 (time 1749542065.70 total 113.44 last 354.76) - open/close 53608 (time 1749542075.70 total 123.44 last 360.79) - open/close 57309 (time 1749542085.70 total 133.44 last 370.08) - open/close 60000 (time 1749542092.61 total 140.35 last 389.54) - open/close 64177 (time 1749542102.61 total 150.35 last 417.68) total: 64768 open/close in 151.75 seconds: 426.80 ops/second - /unlink 7848 (time 1749542114.87 total 10.00 last 784.77) - /unlink 10000 (time 1749542117.40 total 12.53 last 849.42) - /unlink 18760 (time 1749542127.41 total 22.53 last 875.93) - /unlink 27052 (time 1749542137.41 total 32.54 last 829.13) - /unlink 30000 (time 1749542140.68 total 35.81 last 901.20) - /unlink 38677 (time 1749542150.68 total 45.81 last 867.61) - /unlink 47612 (time 1749542160.68 total 55.81 last 893.48) - /unlink 50000 (time 1749542163.34 total 58.47 last 897.50) - /unlink 59427 (time 1749542173.34 total 68.47 last 942.64) total: 64768 /unlink in 74.14 seconds: 873.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 PASS 106 (1367s) Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 oleg424-server: oleg424-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1475 sec ========== 03:57:20 (1749542240) === conf-sanity: start cleanup 03:57:20 (1749542240) === === conf-sanity: finish cleanup 03:57:21 (1749542241) ===