-----============= acceptance-small: conf-sanity ============----- Fri Jul 19 12:19:49 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg249-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg249-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg249-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg249-server oleg249-server: oleg249-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: detected 4 online CPUs by sysfs oleg249-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg249-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg249-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg249-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg249-server: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg249-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg249-server: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg249-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:21:15 (1721406075) Stopping clients: oleg249-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg249-client.virtnet /mnt/lustre2 (opts:-f) oleg249-server: oleg249-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: detected 4 online CPUs by sysfs oleg249-server: Force libcfs to create 2 CPU partitions oleg249-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg249-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg249-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg249-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock oleg249-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4410 (time 1721406118.60 total 10.00 last 440.93) - open/close 8458 (time 1721406128.60 total 20.00 last 404.79) - open/close 10000 (time 1721406132.04 total 23.43 last 449.18) - open/close 14369 (time 1721406142.04 total 33.44 last 436.85) - open/close 19283 (time 1721406152.04 total 43.44 last 491.35) - open/close 25248 (time 1721406162.04 total 53.44 last 596.44) - open/close 30000 (time 1721406170.66 total 62.06 last 551.29) - open/close 35409 (time 1721406180.66 total 72.06 last 540.84) - open/close 39971 (time 1721406190.66 total 82.06 last 456.15) - open/close 44209 (time 1721406200.66 total 92.06 last 423.71) - open/close 48782 (time 1721406210.67 total 102.06 last 457.22) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 103.98 seconds: 477.97 ops/second - /unlink 8841 (time 1721406223.49 total 10.00 last 884.01) - /unlink 16967 (time 1721406233.49 total 20.00 last 812.58) - /unlink 20000 (time 1721406236.89 total 23.40 last 892.97) - /unlink 30000 (time 1721406246.41 total 32.92 last 1050.17) - /unlink 40000 (time 1721406256.02 total 42.53 last 1040.82) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 51.56 seconds: 963.97 ops/second - open/close 4902 (time 1721406275.88 total 10.00 last 490.19) - open/close 9680 (time 1721406285.88 total 20.00 last 477.79) - open/close 15092 (time 1721406295.88 total 30.00 last 541.17) - open/close 19608 (time 1721406305.88 total 40.00 last 451.59) - open/close 24496 (time 1721406315.89 total 50.00 last 488.75) - open/close 29647 (time 1721406325.89 total 60.00 last 515.09) - open/close 34746 (time 1721406335.89 total 70.00 last 509.87) - open/close 40000 (time 1721406345.75 total 79.87 last 532.52) - open/close 45406 (time 1721406355.75 total 89.87 last 540.51) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 97.09 seconds: 511.89 ops/second - /unlink 10000 (time 1721406372.72 total 8.91 last 1122.84) - /unlink 20000 (time 1721406381.07 total 17.26 last 1197.59) - /unlink 30000 (time 1721406389.54 total 25.73 last 1180.36) - /unlink 40000 (time 1721406398.73 total 34.91 last 1088.72) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 43.22 seconds: 1149.98 ops/second - open/close 5549 (time 1721406417.90 total 10.00 last 554.89) - open/close 10000 (time 1721406426.11 total 18.22 last 541.77) - open/close 15454 (time 1721406436.12 total 28.22 last 545.29) - open/close 20000 (time 1721406444.22 total 36.32 last 560.80) - open/close 26357 (time 1721406454.22 total 46.33 last 635.58) - open/close 30000 (time 1721406459.90 total 52.00 last 641.89) - open/close 36277 (time 1721406469.90 total 62.00 last 627.59) - open/close 40000 (time 1721406475.95 total 68.05 last 615.47) - open/close 45751 (time 1721406485.95 total 78.05 last 575.09) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 84.34 seconds: 589.29 ops/second - /unlink 10000 (time 1721406502.56 total 9.51 last 1051.98) - /unlink 20000 (time 1721406512.27 total 19.21 last 1030.56) - /unlink 30000 (time 1721406522.20 total 29.15 last 1006.42) - /unlink 40000 (time 1721406530.87 total 37.82 last 1153.22) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 45.85 seconds: 1083.92 ops/second - open/close 6942 (time 1721406549.68 total 10.00 last 694.19) - open/close 10000 (time 1721406554.05 total 14.37 last 700.32) - open/close 16099 (time 1721406564.05 total 24.37 last 609.81) - open/close 20000 (time 1721406570.55 total 30.87 last 599.71) - open/close 25582 (time 1721406580.55 total 40.87 last 558.14) - open/close 30000 (time 1721406588.42 total 48.74 last 561.51) - open/close 35576 (time 1721406598.42 total 58.74 last 557.52) - open/close 40000 (time 1721406606.91 total 67.23 last 521.39) - open/close 45318 (time 1721406616.91 total 77.23 last 531.75) open(/mnt/lustre/d106.conf-sanity/f-48494) error: No space left on device total: 48494 open/close in 82.91 seconds: 584.90 ops/second - /unlink 10000 (time 1721406633.37 total 9.91 last 1009.28) - /unlink 20000 (time 1721406642.93 total 19.47 last 1045.62) - /unlink 30000 (time 1721406652.26 total 28.81 last 1071.29) - /unlink 40000 (time 1721406661.58 total 38.13 last 1073.06) unlink(/mnt/lustre/d106.conf-sanity/f-48494) error: No such file or directory total: 48494 /unlink in 46.28 seconds: 1047.73 ops/second - open/close 5562 (time 1721406680.59 total 10.00 last 556.11) - open/close 10000 (time 1721406688.77 total 18.18 last 542.64) - open/close 14768 (time 1721406698.77 total 28.18 last 476.73) - open/close 20000 (time 1721406708.21 total 37.62 last 554.45) - open/close 25663 (time 1721406718.21 total 47.62 last 566.22) - open/close 30000 (time 1721406727.25 total 56.66 last 479.61) - open/close 35130 (time 1721406737.25 total 66.66 last 512.99) - open/close 40000 (time 1721406746.92 total 76.33 last 503.76) - open/close 44127 (time 1721406756.92 total 86.33 last 412.68) open(/mnt/lustre/d106.conf-sanity/f-48494) error: No space left on device total: 48494 open/close in 94.50 seconds: 513.15 ops/second - /unlink 9163 (time 1721406776.00 total 10.00 last 916.26) - /unlink 18676 (time 1721406786.00 total 20.00 last 951.22) - /unlink 29083 (time 1721406796.00 total 30.00 last 1040.69) - /unlink 39746 (time 1721406806.00 total 40.00 last 1066.26) unlink(/mnt/lustre/d106.conf-sanity/f-48494) error: No such file or directory total: 48494 /unlink in 48.09 seconds: 1008.30 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server LNET ready to unload unloading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local oleg249-server: LNET ready to unload modules unloaded. pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 PASS 106 (769s) Stopping clients: oleg249-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg249-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 oleg249-server: oleg249-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: detected 4 online CPUs by sysfs oleg249-server: Force libcfs to create 2 CPU partitions oleg249-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg249-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 876 sec =========== 12:34:26 (1721406866)