-----============= acceptance-small: conf-sanity ============----- Mon Jul 29 03:25:10 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg142-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg142-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg142-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg142-server oleg142-server: oleg142-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: detected 4 online CPUs by sysfs oleg142-server: Force libcfs to create 2 CPU partitions oleg142-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' oleg142-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg142-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg142-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg142-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg142-server: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg142-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg142-server: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg142-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg142-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:26:38 (1722237998) Stopping clients: oleg142-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg142-client.virtnet /mnt/lustre2 (opts:-f) oleg142-server: oleg142-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: detected 4 online CPUs by sysfs oleg142-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg142-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg142-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg142-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg142-client.virtnet: -o user_xattr,flock oleg142-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3673 (time 1722238042.07 total 10.00 last 367.27) - open/close 7676 (time 1722238052.07 total 20.00 last 400.25) - open/close 10000 (time 1722238057.58 total 25.51 last 421.73) - open/close 14734 (time 1722238067.58 total 35.51 last 473.35) - open/close 19985 (time 1722238077.58 total 45.52 last 525.02) - open/close 25020 (time 1722238087.58 total 55.52 last 503.48) - open/close 30000 (time 1722238097.39 total 65.33 last 507.54) - open/close 35478 (time 1722238107.40 total 75.33 last 547.71) - open/close 40000 (time 1722238115.67 total 83.60 last 546.70) - open/close 44999 (time 1722238125.67 total 93.60 last 499.80) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 102.91 seconds: 482.92 ops/second - /unlink 9026 (time 1722238145.83 total 10.00 last 902.55) - /unlink 18151 (time 1722238155.83 total 20.00 last 912.47) - /unlink 20000 (time 1722238157.86 total 22.03 last 910.29) - /unlink 28652 (time 1722238167.86 total 32.03 last 865.16) - /unlink 37876 (time 1722238177.87 total 42.04 last 922.04) - /unlink 40000 (time 1722238180.10 total 44.27 last 951.26) - /unlink 49303 (time 1722238190.10 total 54.27 last 930.28) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 54.69 seconds: 908.77 ops/second - open/close 4761 (time 1722238201.39 total 10.01 last 475.71) - open/close 9819 (time 1722238211.39 total 20.01 last 505.80) - open/close 14868 (time 1722238221.39 total 30.01 last 504.85) - open/close 19886 (time 1722238231.39 total 40.01 last 501.79) - open/close 25197 (time 1722238241.39 total 50.01 last 531.09) - open/close 30000 (time 1722238250.47 total 59.08 last 529.26) - open/close 35190 (time 1722238260.47 total 69.09 last 518.92) - open/close 40000 (time 1722238270.34 total 78.96 last 487.15) - open/close 45161 (time 1722238280.34 total 88.96 last 516.09) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 97.69 seconds: 508.71 ops/second - /unlink 9650 (time 1722238299.94 total 10.00 last 964.93) - /unlink 19495 (time 1722238309.94 total 20.00 last 984.48) - /unlink 29048 (time 1722238319.94 total 30.00 last 955.29) - /unlink 38755 (time 1722238329.94 total 40.00 last 970.67) - /unlink 48342 (time 1722238339.94 total 50.00 last 958.64) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 51.49 seconds: 965.16 ops/second - open/close 5260 (time 1722238352.30 total 10.00 last 525.96) - open/close 10000 (time 1722238361.06 total 18.77 last 540.67) - open/close 15181 (time 1722238371.06 total 28.77 last 518.09) - open/close 20000 (time 1722238380.28 total 37.99 last 522.71) - open/close 25212 (time 1722238390.28 total 47.99 last 521.14) - open/close 29984 (time 1722238400.29 total 57.99 last 477.14) - open/close 35237 (time 1722238410.29 total 67.99 last 525.23) - open/close 40000 (time 1722238419.62 total 77.32 last 510.31) - open/close 45229 (time 1722238429.62 total 87.33 last 522.85) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 96.60 seconds: 514.47 ops/second - /unlink 8514 (time 1722238449.82 total 10.00 last 851.38) - /unlink 17521 (time 1722238459.82 total 20.00 last 900.65) - /unlink 20000 (time 1722238462.55 total 22.73 last 909.22) - /unlink 29681 (time 1722238472.55 total 32.73 last 968.09) - /unlink 39473 (time 1722238482.55 total 42.73 last 979.14) - /unlink 49420 (time 1722238492.55 total 52.73 last 994.63) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 53.04 seconds: 937.02 ops/second - open/close 5702 (time 1722238503.71 total 10.00 last 570.18) - open/close 10000 (time 1722238511.43 total 17.72 last 556.87) - open/close 15497 (time 1722238521.43 total 27.72 last 549.63) - open/close 20000 (time 1722238529.21 total 35.50 last 579.04) - open/close 25730 (time 1722238539.21 total 45.50 last 572.95) - open/close 30000 (time 1722238547.54 total 53.83 last 512.56) - open/close 35255 (time 1722238557.54 total 63.83 last 525.48) - open/close 40000 (time 1722238566.11 total 72.40 last 553.30) - open/close 44825 (time 1722238576.11 total 82.41 last 482.43) open(/mnt/lustre/d106.conf-sanity/f-49129) error: No space left on device total: 49129 open/close in 91.33 seconds: 537.90 ops/second - /unlink 5261 (time 1722238595.90 total 10.00 last 525.93) - /unlink 8008 (time 1722238605.90 total 20.01 last 274.62) - /unlink 10000 (time 1722238613.18 total 27.28 last 273.67) - /unlink 14673 (time 1722238623.18 total 37.29 last 467.28) - /unlink 19715 (time 1722238633.18 total 47.29 last 504.18) - /unlink 23962 (time 1722238643.18 total 57.29 last 424.68) - /unlink 27544 (time 1722238653.18 total 67.29 last 358.17) - /unlink 30000 (time 1722238658.63 total 72.73 last 451.20) - /unlink 35769 (time 1722238668.63 total 82.73 last 576.83) - /unlink 40000 (time 1722238674.96 total 89.07 last 667.86) - /unlink 46653 (time 1722238684.96 total 99.07 last 665.20) unlink(/mnt/lustre/d106.conf-sanity/f-49129) error: No such file or directory total: 49129 /unlink in 103.69 seconds: 473.79 ops/second - open/close 3884 (time 1722238700.52 total 10.00 last 388.34) - open/close 6493 (time 1722238710.52 total 20.00 last 260.90) - open/close 9482 (time 1722238720.53 total 30.00 last 298.81) - open/close 12614 (time 1722238730.53 total 40.01 last 313.11) - open/close 16009 (time 1722238740.53 total 50.01 last 339.40) - open/close 18620 (time 1722238750.53 total 60.01 last 261.09) - open/close 20000 (time 1722238755.64 total 65.12 last 270.13) - open/close 22991 (time 1722238765.64 total 75.12 last 299.07) - open/close 26270 (time 1722238775.64 total 85.12 last 327.86) - open/close 29881 (time 1722238785.64 total 95.12 last 361.08) - open/close 33032 (time 1722238795.66 total 105.14 last 314.57) - open/close 36407 (time 1722238805.66 total 115.14 last 337.47) - open/close 39734 (time 1722238815.66 total 125.14 last 332.66) - open/close 42724 (time 1722238825.67 total 135.14 last 298.92) - open/close 45882 (time 1722238835.67 total 145.15 last 315.71) open(/mnt/lustre/d106.conf-sanity/f-49129) error: No space left on device total: 49129 open/close in 154.71 seconds: 317.55 ops/second - /unlink 5816 (time 1722238856.29 total 10.00 last 581.55) - /unlink 10000 (time 1722238863.59 total 17.30 last 573.34) - /unlink 16695 (time 1722238873.59 total 27.30 last 669.44) - /unlink 20000 (time 1722238879.40 total 33.11 last 569.01) - /unlink 27564 (time 1722238889.40 total 43.11 last 756.35) - /unlink 30000 (time 1722238892.72 total 46.43 last 734.38) - /unlink 35851 (time 1722238902.72 total 56.43 last 584.96) - /unlink 40000 (time 1722238909.79 total 63.49 last 587.23) - /unlink 46151 (time 1722238919.79 total 73.49 last 615.03) unlink(/mnt/lustre/d106.conf-sanity/f-49129) error: No such file or directory total: 49129 /unlink in 77.90 seconds: 630.64 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg142-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg142-server LNET ready to unload unloading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing unload_modules_local oleg142-server: LNET ready to unload modules unloaded. pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 PASS 106 (955s) Stopping clients: oleg142-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg142-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 oleg142-server: oleg142-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: detected 4 online CPUs by sysfs oleg142-server: Force libcfs to create 2 CPU partitions oleg142-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg142-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1065 sec ========== 03:42:55 (1722238975)