-----============= acceptance-small: conf-sanity ============----- Wed May 7 09:45:37 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg151-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg151-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg151-server oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg151-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg151-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:46:30 (1746625590) Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg151-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32587 331 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5481 (time 1746625624.24 total 10.00 last 548.04) - open/close 10000 (time 1746625632.79 total 18.55 last 528.31) - open/close 15208 (time 1746625642.79 total 28.56 last 520.72) - open/close 20000 (time 1746625651.53 total 37.30 last 548.28) - open/close 25158 (time 1746625661.53 total 47.30 last 515.79) - open/close 28355 (time 1746625671.54 total 57.30 last 319.65) - open/close 29681 (time 1746625681.55 total 67.31 last 132.44) - open/close 30000 (time 1746625684.57 total 70.33 last 105.57) - open/close 31382 (time 1746625694.57 total 80.34 last 138.14) - open/close 33613 (time 1746625704.58 total 90.34 last 222.98) - open/close 36393 (time 1746625714.58 total 100.34 last 277.92) - open/close 38514 (time 1746625724.59 total 110.35 last 211.98) - open/close 40000 (time 1746625733.43 total 119.19 last 168.03) - open/close 42699 (time 1746625743.43 total 129.19 last 269.89) - open/close 46673 (time 1746625753.43 total 139.20 last 397.31) - open/close 50000 (time 1746625760.88 total 146.64 last 446.83) - open/close 52907 (time 1746625770.88 total 156.64 last 290.69) - open/close 55607 (time 1746625780.88 total 166.64 last 269.97) - open/close 58549 (time 1746625790.88 total 176.65 last 294.15) - open/close 60000 (time 1746625795.22 total 180.98 last 334.86) - open/close 64089 (time 1746625805.22 total 190.98 last 408.83) total: 64768 open/close in 193.39 seconds: 334.90 ops/second - /unlink 4950 (time 1746625818.58 total 10.00 last 494.98) - /unlink 10000 (time 1746625825.17 total 16.59 last 766.22) - /unlink 17066 (time 1746625835.17 total 26.59 last 706.57) - /unlink 20000 (time 1746625839.40 total 30.82 last 693.57) - /unlink 26983 (time 1746625849.40 total 40.82 last 698.26) - /unlink 30000 (time 1746625853.23 total 44.65 last 788.79) - /unlink 39418 (time 1746625863.23 total 54.65 last 941.74) - /unlink 48834 (time 1746625873.23 total 64.65 last 941.58) - /unlink 58227 (time 1746625883.23 total 74.65 last 939.23) - /unlink 60000 (time 1746625885.26 total 76.68 last 874.44) total: 64768 /unlink in 82.38 seconds: 786.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5137 (time 1746625921.89 total 10.00 last 513.61) - open/close 9860 (time 1746625931.90 total 20.00 last 472.18) - open/close 13392 (time 1746625941.90 total 30.01 last 353.14) - open/close 18437 (time 1746625951.90 total 40.01 last 504.50) - open/close 20000 (time 1746625955.03 total 43.13 last 499.57) - open/close 24781 (time 1746625965.03 total 53.14 last 478.01) - open/close 29561 (time 1746625975.03 total 63.14 last 477.98) - open/close 34107 (time 1746625985.03 total 73.14 last 454.53) - open/close 38905 (time 1746625995.03 total 83.14 last 479.79) - open/close 44488 (time 1746626005.03 total 93.14 last 558.18) - open/close 49574 (time 1746626015.04 total 103.14 last 508.56) - open/close 54180 (time 1746626025.04 total 113.14 last 460.59) - open/close 59023 (time 1746626035.04 total 123.14 last 484.16) - open/close 60000 (time 1746626037.93 total 126.04 last 337.75) total: 64768 open/close in 135.47 seconds: 478.11 ops/second - /unlink 8758 (time 1746626058.16 total 10.00 last 875.71) - /unlink 16291 (time 1746626068.16 total 20.00 last 753.22) - /unlink 20000 (time 1746626073.49 total 25.33 last 695.97) - /unlink 26001 (time 1746626083.49 total 35.33 last 600.05) - /unlink 30000 (time 1746626090.32 total 42.17 last 585.06) - /unlink 36399 (time 1746626100.32 total 52.17 last 639.82) - /unlink 40000 (time 1746626106.67 total 58.51 last 567.70) - /unlink 46043 (time 1746626116.67 total 68.51 last 604.24) - /unlink 50000 (time 1746626123.43 total 75.28 last 584.99) - /unlink 58170 (time 1746626133.43 total 85.28 last 816.98) - /unlink 60000 (time 1746626135.51 total 87.35 last 881.27) total: 64768 /unlink in 94.50 seconds: 685.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3711 (time 1746626173.26 total 10.00 last 371.03) - open/close 7185 (time 1746626183.26 total 20.00 last 347.37) - open/close 10000 (time 1746626190.70 total 27.44 last 378.68) - open/close 13607 (time 1746626200.70 total 37.44 last 360.59) - open/close 17744 (time 1746626210.70 total 47.44 last 413.70) - open/close 20000 (time 1746626214.41 total 51.15 last 607.77) - open/close 24066 (time 1746626224.42 total 61.16 last 406.41) - open/close 28039 (time 1746626234.42 total 71.16 last 397.24) - open/close 30000 (time 1746626238.03 total 74.77 last 542.60) - open/close 34777 (time 1746626248.03 total 84.77 last 477.69) - open/close 39576 (time 1746626258.03 total 94.77 last 479.84) - open/close 43431 (time 1746626268.04 total 104.78 last 385.42) - open/close 47069 (time 1746626278.04 total 114.78 last 363.75) - open/close 50000 (time 1746626283.91 total 120.65 last 499.21) - open/close 55137 (time 1746626293.91 total 130.65 last 513.64) - open/close 59538 (time 1746626303.91 total 140.65 last 440.04) - open/close 63154 (time 1746626313.92 total 150.65 last 361.48) total: 64768 open/close in 155.76 seconds: 415.83 ops/second - /unlink 6373 (time 1746626330.02 total 10.00 last 637.29) - /unlink 10000 (time 1746626334.77 total 14.75 last 764.25) - /unlink 17630 (time 1746626344.77 total 24.75 last 762.94) - /unlink 20000 (time 1746626348.58 total 28.56 last 620.99) - /unlink 26441 (time 1746626358.59 total 38.56 last 644.07) - /unlink 30000 (time 1746626363.55 total 43.53 last 716.21) - /unlink 37530 (time 1746626373.56 total 53.53 last 752.97) - /unlink 40000 (time 1746626376.71 total 56.69 last 782.78) - /unlink 48089 (time 1746626386.71 total 66.69 last 808.84) - /unlink 50000 (time 1746626389.11 total 69.09 last 797.62) - /unlink 58128 (time 1746626399.11 total 79.09 last 812.74) - /unlink 60000 (time 1746626401.27 total 81.25 last 865.27) total: 64768 /unlink in 86.48 seconds: 748.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5232 (time 1746626438.19 total 10.00 last 523.08) - open/close 10000 (time 1746626446.48 total 18.29 last 575.22) - open/close 15754 (time 1746626456.48 total 28.29 last 575.38) - open/close 20000 (time 1746626463.81 total 35.63 last 578.87) - open/close 25922 (time 1746626473.81 total 45.63 last 592.14) - open/close 30000 (time 1746626480.51 total 52.32 last 609.01) - open/close 35507 (time 1746626490.51 total 62.32 last 550.63) - open/close 40000 (time 1746626499.53 total 71.35 last 497.97) - open/close 45462 (time 1746626509.53 total 81.35 last 546.18) - open/close 50000 (time 1746626517.22 total 89.04 last 590.27) - open/close 55964 (time 1746626527.22 total 99.04 last 596.40) - open/close 60000 (time 1746626534.15 total 105.96 last 582.71) total: 64768 open/close in 114.17 seconds: 567.32 ops/second - /unlink 8413 (time 1746626553.13 total 10.00 last 841.26) - /unlink 17223 (time 1746626563.13 total 20.00 last 880.96) - /unlink 20000 (time 1746626566.22 total 23.08 last 900.58) - /unlink 28902 (time 1746626576.22 total 33.08 last 890.16) - /unlink 37456 (time 1746626586.22 total 43.09 last 855.31) - /unlink 40000 (time 1746626589.21 total 46.08 last 848.68) - /unlink 48574 (time 1746626599.22 total 56.08 last 857.33) - /unlink 56898 (time 1746626609.22 total 66.08 last 832.40) - /unlink 60000 (time 1746626612.87 total 69.74 last 848.82) total: 64768 /unlink in 75.61 seconds: 856.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5013 (time 1746626653.75 total 10.00 last 501.21) - open/close 10000 (time 1746626663.68 total 19.93 last 502.22) - open/close 15201 (time 1746626673.68 total 29.94 last 519.87) - open/close 20000 (time 1746626683.59 total 39.84 last 484.53) - open/close 22388 (time 1746626693.59 total 49.84 last 238.76) - open/close 24651 (time 1746626703.59 total 59.84 last 226.27) - open/close 28538 (time 1746626713.59 total 69.84 last 388.69) - open/close 30000 (time 1746626716.56 total 72.81 last 492.98) - open/close 34288 (time 1746626726.56 total 82.81 last 428.74) - open/close 37183 (time 1746626736.56 total 92.81 last 289.45) - open/close 40000 (time 1746626743.97 total 100.22 last 380.43) - open/close 44564 (time 1746626753.97 total 110.22 last 456.38) - open/close 48992 (time 1746626763.97 total 120.22 last 442.73) - open/close 50000 (time 1746626766.17 total 122.42 last 457.81) - open/close 54933 (time 1746626776.17 total 132.42 last 493.23) - open/close 57832 (time 1746626786.17 total 142.42 last 289.83) - open/close 60000 (time 1746626793.81 total 150.06 last 283.95) - open/close 63879 (time 1746626803.81 total 160.06 last 387.87) total: 64768 open/close in 162.48 seconds: 398.62 ops/second - /unlink 10000 (time 1746626816.98 total 9.88 last 1011.91) - /unlink 19187 (time 1746626826.98 total 19.88 last 918.68) - /unlink 28277 (time 1746626836.98 total 29.88 last 908.93) - /unlink 36566 (time 1746626846.98 total 39.88 last 828.85) - /unlink 40000 (time 1746626851.21 total 44.11 last 811.77) - /unlink 48669 (time 1746626861.21 total 54.12 last 866.83) - /unlink 56759 (time 1746626871.21 total 64.12 last 808.96) - /unlink 60000 (time 1746626875.04 total 67.94 last 847.55) total: 64768 /unlink in 73.31 seconds: 883.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 PASS 106 (1338s) Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1409 sec ========== 10:09:06 (1746626946) === conf-sanity: start cleanup 10:09:06 (1746626946) === === conf-sanity: finish cleanup 10:09:07 (1746626947) ===