-----============= acceptance-small: conf-sanity ============----- Sat May 11 16:54:21 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg234-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg234-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg234-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg234-server oleg234-server: oleg234-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg234-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg234-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg234-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg234-server: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg234-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg234-server: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg234-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:55:43 (1715460943) Stopping clients: oleg234-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg234-client.virtnet /mnt/lustre2 (opts:-f) oleg234-server: oleg234-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg234-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg234-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg234-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock oleg234-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5681 (time 1715460982.92 total 10.00 last 568.08) - open/close 10000 (time 1715460990.48 total 17.56 last 571.67) - open/close 15859 (time 1715461000.48 total 27.56 last 585.83) - open/close 20000 (time 1715461007.41 total 34.49 last 597.63) - open/close 26129 (time 1715461017.41 total 44.49 last 612.86) - open/close 30000 (time 1715461023.68 total 50.75 last 617.64) - open/close 35986 (time 1715461033.68 total 60.75 last 598.60) - open/close 40000 (time 1715461040.56 total 67.63 last 583.46) - open/close 45533 (time 1715461050.56 total 77.63 last 553.22) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 84.96 seconds: 584.95 ops/second - /unlink 10000 (time 1715461067.83 total 9.11 last 1098.15) - /unlink 20000 (time 1715461076.67 total 17.95 last 1131.30) - /unlink 30000 (time 1715461085.54 total 26.82 last 1127.12) - /unlink 40000 (time 1715461094.56 total 35.84 last 1108.99) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 44.48 seconds: 1117.37 ops/second - open/close 5398 (time 1715461114.07 total 10.00 last 539.80) - open/close 10000 (time 1715461121.06 total 16.98 last 658.86) - open/close 17125 (time 1715461131.06 total 26.99 last 712.47) - open/close 20000 (time 1715461135.06 total 30.99 last 718.06) - open/close 27091 (time 1715461145.06 total 40.99 last 709.07) - open/close 30000 (time 1715461149.93 total 45.85 last 598.27) - open/close 35993 (time 1715461159.93 total 55.85 last 599.21) - open/close 40000 (time 1715461166.60 total 62.53 last 600.30) - open/close 45878 (time 1715461176.60 total 72.53 last 587.73) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 79.12 seconds: 628.10 ops/second - /unlink 10000 (time 1715461193.24 total 9.19 last 1087.60) - /unlink 20000 (time 1715461202.32 total 18.28 last 1100.80) - /unlink 30000 (time 1715461211.59 total 27.55 last 1078.60) - /unlink 40000 (time 1715461220.80 total 36.75 last 1086.41) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 45.15 seconds: 1100.78 ops/second - open/close 6172 (time 1715461240.00 total 10.00 last 617.18) - open/close 10000 (time 1715461246.16 total 16.16 last 621.57) - open/close 16169 (time 1715461256.16 total 26.16 last 616.81) - open/close 20000 (time 1715461262.37 total 32.37 last 616.82) - open/close 26130 (time 1715461272.37 total 42.37 last 612.98) - open/close 30000 (time 1715461278.65 total 48.66 last 615.91) - open/close 36230 (time 1715461288.66 total 58.66 last 622.91) - open/close 40000 (time 1715461294.70 total 64.70 last 623.54) - open/close 46171 (time 1715461304.70 total 74.70 last 617.01) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 80.40 seconds: 618.13 ops/second - /unlink 10000 (time 1715461320.08 total 8.86 last 1128.06) - /unlink 20000 (time 1715461328.96 total 17.74 last 1126.13) - /unlink 30000 (time 1715461337.71 total 26.49 last 1142.96) - /unlink 40000 (time 1715461346.34 total 35.13 last 1158.10) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 43.09 seconds: 1153.28 ops/second - open/close 5546 (time 1715461365.10 total 10.00 last 554.58) - open/close 10000 (time 1715461374.70 total 19.60 last 463.88) - open/close 14118 (time 1715461384.70 total 29.60 last 411.71) - open/close 18110 (time 1715461394.71 total 39.61 last 399.07) - open/close 20000 (time 1715461401.06 total 45.96 last 297.39) - open/close 24161 (time 1715461411.06 total 55.96 last 416.07) - open/close 28180 (time 1715461421.06 total 65.96 last 401.89) - open/close 30000 (time 1715461425.79 total 70.69 last 385.13) - open/close 33724 (time 1715461435.79 total 80.69 last 372.34) - open/close 37274 (time 1715461445.79 total 90.69 last 354.96) - open/close 40000 (time 1715461452.25 total 97.15 last 422.34) - open/close 44021 (time 1715461462.25 total 107.15 last 402.10) open(/mnt/lustre/d106.conf-sanity/f-47488) error: No space left on device total: 47488 open/close in 116.53 seconds: 407.50 ops/second - /unlink 8839 (time 1715461482.63 total 10.00 last 883.87) - /unlink 16409 (time 1715461492.63 total 20.00 last 756.99) - /unlink 20000 (time 1715461498.64 total 26.01 last 597.34) - /unlink 27862 (time 1715461508.64 total 36.01 last 786.20) - /unlink 30000 (time 1715461512.54 total 39.91 last 548.38) - /unlink 36885 (time 1715461522.54 total 49.91 last 688.48) - /unlink 40000 (time 1715461527.93 total 55.30 last 578.14) - /unlink 46222 (time 1715461537.93 total 65.30 last 622.19) unlink(/mnt/lustre/d106.conf-sanity/f-47488) error: No such file or directory total: 47488 /unlink in 67.85 seconds: 699.90 ops/second - open/close 2758 (time 1715461551.72 total 10.00 last 275.79) - open/close 6165 (time 1715461561.72 total 20.00 last 340.59) - open/close 9159 (time 1715461571.72 total 30.00 last 299.38) - open/close 10000 (time 1715461574.74 total 33.02 last 279.00) - open/close 12801 (time 1715461584.74 total 43.02 last 280.05) - open/close 16117 (time 1715461594.74 total 53.02 last 331.55) - open/close 19625 (time 1715461604.74 total 63.02 last 350.72) - open/close 22540 (time 1715461614.75 total 73.03 last 291.44) - open/close 25790 (time 1715461624.75 total 83.03 last 324.91) - open/close 28710 (time 1715461634.75 total 93.03 last 291.93) - open/close 30000 (time 1715461639.55 total 97.84 last 268.52) - open/close 33114 (time 1715461649.56 total 107.84 last 311.28) - open/close 36660 (time 1715461659.56 total 117.84 last 354.49) - open/close 39649 (time 1715461669.56 total 127.84 last 298.89) - open/close 42317 (time 1715461679.56 total 137.84 last 266.78) - open/close 45017 (time 1715461689.57 total 147.85 last 269.93) open(/mnt/lustre/d106.conf-sanity/f-47488) error: No space left on device total: 47488 open/close in 157.27 seconds: 301.95 ops/second - /unlink 5030 (time 1715461710.26 total 10.00 last 502.97) - /unlink 10000 (time 1715461719.83 total 19.57 last 519.40) - /unlink 14918 (time 1715461729.83 total 29.57 last 491.72) - /unlink 20000 (time 1715461739.75 total 39.49 last 512.56) - /unlink 25697 (time 1715461749.75 total 49.49 last 569.59) - /unlink 30000 (time 1715461757.07 total 56.81 last 587.52) - /unlink 37686 (time 1715461767.07 total 66.81 last 768.51) - /unlink 40000 (time 1715461771.43 total 71.17 last 531.24) - /unlink 45496 (time 1715461781.43 total 81.17 last 549.53) unlink(/mnt/lustre/d106.conf-sanity/f-47488) error: No such file or directory total: 47488 /unlink in 84.03 seconds: 565.10 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server LNET ready to unload unloading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: LNET ready to unload modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 PASS 106 (872s) Stopping clients: oleg234-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg234-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 oleg234-server: oleg234-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 980 sec =========== 17:10:41 (1715461841)