-----============= acceptance-small: conf-sanity ============----- Fri May 31 06:54:50 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg240-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg240-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg240-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg240-server oleg240-server: oleg240-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: detected 4 online CPUs by sysfs oleg240-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg240-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg240-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg240-server: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg240-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg240-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server stop mds service on oleg240-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:55:53 (1717152953) Stopping clients: oleg240-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg240-client.virtnet /mnt/lustre2 (opts:-f) oleg240-server: oleg240-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: detected 4 online CPUs by sysfs oleg240-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg240-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg240-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg240-client.virtnet: -o user_xattr,flock oleg240-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5143 (time 1717152989.04 total 10.00 last 514.25) - open/close 10000 (time 1717152998.63 total 19.60 last 506.07) - open/close 14688 (time 1717153008.63 total 29.60 last 468.80) - open/close 19696 (time 1717153018.63 total 39.60 last 500.79) - open/close 25277 (time 1717153028.63 total 49.60 last 558.06) - open/close 30000 (time 1717153037.05 total 58.01 last 561.39) - open/close 35934 (time 1717153047.05 total 68.01 last 593.39) - open/close 40000 (time 1717153054.01 total 74.98 last 583.84) - open/close 45795 (time 1717153064.01 total 84.98 last 579.48) - open/close 50000 (time 1717153071.55 total 92.52 last 557.82) - open/close 55870 (time 1717153081.55 total 102.52 last 586.97) - open/close 60000 (time 1717153089.00 total 109.97 last 554.08) total: 64768 open/close in 118.28 seconds: 547.58 ops/second - /unlink 9864 (time 1717153108.08 total 10.00 last 986.31) - /unlink 19788 (time 1717153118.08 total 20.00 last 992.31) - /unlink 29205 (time 1717153128.08 total 30.00 last 941.65) - /unlink 38485 (time 1717153138.08 total 40.00 last 927.95) - /unlink 48124 (time 1717153148.08 total 50.00 last 963.88) - /unlink 58149 (time 1717153158.08 total 60.00 last 1002.43) total: 64768 /unlink in 66.06 seconds: 980.41 ops/second - open/close 6026 (time 1717153174.93 total 10.00 last 602.52) - open/close 10000 (time 1717153181.44 total 16.52 last 609.65) - open/close 15959 (time 1717153191.45 total 26.52 last 595.79) - open/close 20000 (time 1717153198.44 total 33.52 last 577.52) - open/close 24807 (time 1717153208.44 total 43.52 last 480.69) - open/close 29564 (time 1717153218.44 total 53.52 last 475.67) - open/close 34298 (time 1717153228.45 total 63.52 last 473.31) - open/close 39184 (time 1717153238.45 total 73.52 last 488.59) - open/close 43895 (time 1717153248.45 total 83.52 last 471.10) - open/close 48906 (time 1717153258.45 total 93.52 last 501.01) - open/close 50000 (time 1717153260.89 total 95.96 last 448.14) - open/close 54996 (time 1717153270.89 total 105.97 last 499.55) - open/close 60000 (time 1717153279.14 total 114.22 last 606.46) total: 64768 open/close in 122.30 seconds: 529.59 ops/second - /unlink 9029 (time 1717153298.06 total 10.00 last 902.88) - /unlink 18174 (time 1717153308.06 total 20.00 last 914.45) - /unlink 27434 (time 1717153318.06 total 30.00 last 925.98) - /unlink 30000 (time 1717153320.79 total 32.73 last 938.73) - /unlink 39305 (time 1717153330.79 total 42.73 last 930.46) - /unlink 47821 (time 1717153340.79 total 52.74 last 851.48) - /unlink 50000 (time 1717153343.22 total 55.16 last 898.00) - /unlink 59021 (time 1717153353.22 total 65.16 last 902.06) total: 64768 /unlink in 71.40 seconds: 907.17 ops/second - open/close 5061 (time 1717153370.29 total 10.00 last 506.06) - open/close 10000 (time 1717153379.42 total 19.13 last 540.95) - open/close 15294 (time 1717153389.42 total 29.13 last 529.25) - open/close 20000 (time 1717153398.12 total 37.83 last 540.97) - open/close 24938 (time 1717153408.12 total 47.83 last 493.73) - open/close 30000 (time 1717153417.77 total 57.48 last 524.81) - open/close 35264 (time 1717153427.77 total 67.48 last 526.34) - open/close 40000 (time 1717153436.75 total 76.46 last 527.68) - open/close 45446 (time 1717153446.75 total 86.46 last 544.57) - open/close 50000 (time 1717153455.30 total 95.01 last 532.68) - open/close 55165 (time 1717153465.30 total 105.01 last 516.42) - open/close 60000 (time 1717153474.42 total 114.13 last 529.87) total: 64768 open/close in 123.16 seconds: 525.89 ops/second - /unlink 9020 (time 1717153494.30 total 10.00 last 901.90) - /unlink 18064 (time 1717153504.30 total 20.00 last 904.32) - /unlink 20000 (time 1717153506.32 total 22.03 last 956.11) - /unlink 29330 (time 1717153516.32 total 32.03 last 932.99) - /unlink 38511 (time 1717153526.32 total 42.03 last 918.09) - /unlink 47412 (time 1717153536.32 total 52.03 last 890.03) - /unlink 50000 (time 1717153539.19 total 54.89 last 903.98) - /unlink 59153 (time 1717153549.19 total 64.89 last 915.23) total: 64768 /unlink in 70.94 seconds: 912.96 ops/second - open/close 5303 (time 1717153566.08 total 10.00 last 530.21) - open/close 10000 (time 1717153574.90 total 18.83 last 532.27) - open/close 14892 (time 1717153584.90 total 28.83 last 489.17) - open/close 19737 (time 1717153594.94 total 38.87 last 482.51) - open/close 24686 (time 1717153604.96 total 48.89 last 493.83) - open/close 29627 (time 1717153614.97 total 58.89 last 494.02) - open/close 34617 (time 1717153624.97 total 68.89 last 498.94) - open/close 39537 (time 1717153634.97 total 78.89 last 491.97) - open/close 44598 (time 1717153644.97 total 88.89 last 506.02) - open/close 49468 (time 1717153654.97 total 98.90 last 486.94) - open/close 54441 (time 1717153664.97 total 108.90 last 497.29) - open/close 59481 (time 1717153674.97 total 118.90 last 503.97) total: 64768 open/close in 128.72 seconds: 503.19 ops/second - /unlink 9181 (time 1717153695.65 total 10.00 last 918.06) - /unlink 18368 (time 1717153705.65 total 20.00 last 918.70) - /unlink 27223 (time 1717153715.65 total 30.00 last 885.47) - /unlink 30000 (time 1717153718.69 total 33.04 last 913.74) - /unlink 38910 (time 1717153728.69 total 43.04 last 890.97) - /unlink 48207 (time 1717153738.69 total 53.04 last 929.65) - /unlink 57483 (time 1717153748.69 total 63.04 last 927.55) - /unlink 60000 (time 1717153751.27 total 65.62 last 976.29) total: 64768 /unlink in 70.65 seconds: 916.70 ops/second - open/close 5277 (time 1717153767.11 total 10.00 last 527.62) - open/close 10000 (time 1717153775.76 total 18.64 last 546.45) - open/close 15538 (time 1717153785.76 total 28.65 last 553.73) - open/close 20000 (time 1717153793.99 total 36.88 last 541.73) - open/close 25456 (time 1717153804.00 total 46.88 last 545.52) - open/close 30000 (time 1717153812.36 total 55.25 last 543.24) - open/close 35331 (time 1717153822.36 total 65.25 last 533.07) - open/close 40000 (time 1717153831.86 total 74.75 last 491.60) - open/close 44145 (time 1717153841.86 total 84.75 last 414.46) - open/close 47473 (time 1717153851.86 total 94.75 last 332.76) - open/close 50000 (time 1717153858.88 total 101.77 last 359.85) - open/close 53921 (time 1717153868.88 total 111.77 last 392.09) - open/close 57913 (time 1717153878.88 total 121.77 last 399.16) - open/close 60000 (time 1717153882.96 total 125.84 last 512.51) total: 64768 open/close in 135.14 seconds: 479.26 ops/second - /unlink 9275 (time 1717153903.05 total 10.00 last 927.46) - /unlink 18683 (time 1717153913.05 total 20.00 last 940.79) - /unlink 28177 (time 1717153923.06 total 30.00 last 949.34) - /unlink 37504 (time 1717153933.06 total 40.00 last 932.63) - /unlink 40000 (time 1717153935.77 total 42.71 last 920.08) - /unlink 49364 (time 1717153945.77 total 52.72 last 936.37) - /unlink 58742 (time 1717153955.77 total 62.72 last 937.75) total: 64768 /unlink in 69.21 seconds: 935.77 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg240-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg240-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server stop mds service on oleg240-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server LNET ready to unload unloading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing unload_modules_local oleg240-server: LNET ready to unload modules unloaded. pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 PASS 106 (1053s) Stopping clients: oleg240-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg240-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 oleg240-server: oleg240-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: detected 4 online CPUs by sysfs oleg240-server: Force libcfs to create 2 CPU partitions oleg240-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg240-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1134 sec ========== 07:13:45 (1717154025)