-----============= acceptance-small: conf-sanity ============----- Tue Nov 5 10:48:47 EST 2024 excepting tests: 32newtarball 110 Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg152-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg152-server oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg152-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg152-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg152-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg152-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:50:25 (1730821825) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg152-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg152-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg152-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock oleg152-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3010 (time 1730821867.78 total 10.00 last 300.99) - open/close 6116 (time 1730821877.78 total 20.00 last 310.59) - open/close 9464 (time 1730821887.78 total 30.00 last 334.76) - open/close 12205 (time 1730821897.79 total 40.01 last 273.95) - open/close 15814 (time 1730821907.79 total 50.01 last 360.83) - open/close 19941 (time 1730821917.79 total 60.01 last 412.66) - open/close 23507 (time 1730821927.79 total 70.01 last 356.51) - open/close 27799 (time 1730821937.80 total 80.01 last 429.18) - open/close 30000 (time 1730821942.57 total 84.79 last 461.22) - open/close 34279 (time 1730821952.57 total 94.79 last 427.85) - open/close 38776 (time 1730821962.57 total 104.79 last 449.69) - open/close 40000 (time 1730821965.52 total 107.74 last 414.67) - open/close 44498 (time 1730821975.52 total 117.74 last 449.79) - open/close 48676 (time 1730821985.52 total 127.74 last 417.78) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 129.93 seconds: 382.49 ops/second - /unlink 6542 (time 1730821998.60 total 10.00 last 654.07) - /unlink 10000 (time 1730822004.76 total 16.16 last 561.59) - /unlink 17031 (time 1730822014.76 total 26.16 last 703.04) - /unlink 20000 (time 1730822018.59 total 29.99 last 774.98) - /unlink 27737 (time 1730822028.60 total 40.00 last 772.73) - /unlink 30000 (time 1730822031.55 total 42.95 last 767.50) - /unlink 35603 (time 1730822041.55 total 52.95 last 560.29) - /unlink 40000 (time 1730822046.99 total 58.39 last 808.10) - /unlink 48370 (time 1730822056.99 total 68.40 last 836.90) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 69.95 seconds: 710.51 ops/second - open/close 5493 (time 1730822069.38 total 10.00 last 549.28) - open/close 10000 (time 1730822077.41 total 18.03 last 561.45) - open/close 15284 (time 1730822087.41 total 28.03 last 528.34) - open/close 20000 (time 1730822096.85 total 37.47 last 499.55) - open/close 25207 (time 1730822106.85 total 47.47 last 520.62) - open/close 29947 (time 1730822116.85 total 57.47 last 474.00) - open/close 33651 (time 1730822126.86 total 67.47 last 370.32) - open/close 37159 (time 1730822136.86 total 77.47 last 350.77) - open/close 40000 (time 1730822143.33 total 83.95 last 438.77) - open/close 44629 (time 1730822153.33 total 93.95 last 462.88) - open/close 48383 (time 1730822163.33 total 103.95 last 375.39) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 107.81 seconds: 460.99 ops/second - /unlink 8275 (time 1730822178.16 total 10.00 last 827.50) - /unlink 10000 (time 1730822180.29 total 12.13 last 810.23) - /unlink 17702 (time 1730822190.29 total 22.13 last 770.12) - /unlink 20000 (time 1730822193.90 total 25.74 last 637.22) - /unlink 26415 (time 1730822203.90 total 35.74 last 641.48) - /unlink 30000 (time 1730822208.50 total 40.33 last 780.15) - /unlink 37174 (time 1730822218.50 total 50.33 last 717.36) - /unlink 40000 (time 1730822221.65 total 53.49 last 895.05) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 63.31 seconds: 785.03 ops/second - open/close 5161 (time 1730822242.32 total 10.00 last 516.06) - open/close 9700 (time 1730822252.32 total 20.00 last 453.81) - open/close 14490 (time 1730822262.33 total 30.00 last 478.91) - open/close 18960 (time 1730822272.33 total 40.01 last 446.97) - open/close 20000 (time 1730822274.42 total 42.10 last 496.49) - open/close 24526 (time 1730822284.42 total 52.10 last 452.58) - open/close 29458 (time 1730822294.42 total 62.10 last 493.17) - open/close 33494 (time 1730822304.42 total 72.10 last 403.58) - open/close 38276 (time 1730822314.42 total 82.10 last 478.18) - open/close 40000 (time 1730822317.51 total 85.19 last 558.90) - open/close 44996 (time 1730822327.52 total 95.19 last 499.19) - open/close 49448 (time 1730822337.52 total 105.20 last 445.09) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 105.73 seconds: 470.05 ops/second - /unlink 9097 (time 1730822348.94 total 10.00 last 909.68) - /unlink 20000 (time 1730822358.90 total 19.96 last 1094.94) - /unlink 30000 (time 1730822367.92 total 28.98 last 1108.62) - /unlink 40000 (time 1730822377.30 total 38.35 last 1066.75) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 47.63 seconds: 1043.38 ops/second - open/close 5383 (time 1730822397.40 total 10.00 last 538.28) - open/close 10000 (time 1730822405.72 total 18.32 last 554.99) - open/close 15030 (time 1730822415.72 total 28.32 last 502.97) - open/close 19812 (time 1730822425.72 total 38.32 last 478.18) - open/close 24510 (time 1730822435.72 total 48.32 last 469.78) - open/close 29319 (time 1730822445.72 total 58.32 last 480.83) - open/close 34830 (time 1730822455.72 total 68.32 last 551.10) - open/close 40000 (time 1730822464.62 total 77.22 last 581.22) - open/close 44673 (time 1730822474.62 total 87.22 last 467.28) open(/mnt/lustre/d106.conf-sanity/f-47324) error: No space left on device total: 47324 open/close in 93.14 seconds: 508.12 ops/second - /unlink 9268 (time 1730822491.48 total 10.00 last 926.73) - /unlink 19089 (time 1730822501.48 total 20.00 last 982.10) - /unlink 28573 (time 1730822511.48 total 30.00 last 948.30) - /unlink 38470 (time 1730822521.48 total 40.00 last 989.62) unlink(/mnt/lustre/d106.conf-sanity/f-47324) error: No such file or directory total: 47324 /unlink in 48.61 seconds: 973.46 ops/second - open/close 5397 (time 1730822540.94 total 10.00 last 539.70) - open/close 10000 (time 1730822549.09 total 18.16 last 564.31) - open/close 15288 (time 1730822559.09 total 28.16 last 528.71) - open/close 20000 (time 1730822568.13 total 37.20 last 521.32) - open/close 25256 (time 1730822578.13 total 47.20 last 525.51) - open/close 30000 (time 1730822587.18 total 56.24 last 524.70) - open/close 35408 (time 1730822597.18 total 66.24 last 540.72) - open/close 40000 (time 1730822605.66 total 74.73 last 541.28) - open/close 45521 (time 1730822615.66 total 84.73 last 552.04) open(/mnt/lustre/d106.conf-sanity/f-47324) error: No space left on device total: 47324 open/close in 88.38 seconds: 535.47 ops/second - /unlink 10000 (time 1730822629.62 total 9.47 last 1055.70) - /unlink 20000 (time 1730822639.04 total 18.90 last 1061.11) - /unlink 30000 (time 1730822648.17 total 28.03 last 1094.70) - /unlink 40000 (time 1730822657.26 total 37.11 last 1101.15) unlink(/mnt/lustre/d106.conf-sanity/f-47324) error: No such file or directory total: 47324 /unlink in 44.01 seconds: 1075.41 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server LNET ready to unload unloading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: LNET ready to unload modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 PASS 106 (867s) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 983 sec =========== 11:05:11 (1730822711)