-----============= acceptance-small: conf-sanity ============----- Wed Jan 15 20:22:58 EST 2025 excepting tests: 32newtarball 110 Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg205-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg205-server oleg205-server: oleg205-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: detected 4 online CPUs by sysfs oleg205-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg205-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg205-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg205-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg205-server: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg205-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg205-server: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg205-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:24:24 (1736990664) Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f) oleg205-server: oleg205-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: detected 4 online CPUs by sysfs oleg205-server: Force libcfs to create 2 CPU partitions oleg205-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg205-server: insmod: ERROR: could not insert module /home/green/git/lustre-release/lustre/../lnet/lnet/lnet.ko: File exists pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg205-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg205-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg205-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5208 (time 1736990704.63 total 10.00 last 520.74) - open/close 10000 (time 1736990713.01 total 18.38 last 571.72) - open/close 15740 (time 1736990723.01 total 28.38 last 574.00) - open/close 20000 (time 1736990729.90 total 35.28 last 617.80) - open/close 25640 (time 1736990739.91 total 45.28 last 563.94) - open/close 30000 (time 1736990747.96 total 53.34 last 541.06) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 58.19 seconds: 560.10 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9937 (time 1736990763.79 total 10.00 last 993.64) - /unlink 19965 (time 1736990773.79 total 20.00 last 1002.75) - /unlink 29540 (time 1736990783.79 total 30.00 last 957.40) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 33.27 seconds: 979.79 ops/second Waiting for MDT destroys to complete - open/close 5525 (time 1736990824.11 total 10.00 last 552.43) - open/close 10000 (time 1736990832.21 total 18.10 last 552.57) - open/close 15568 (time 1736990842.21 total 28.10 last 556.70) - open/close 20000 (time 1736990850.18 total 36.06 last 556.56) - open/close 25381 (time 1736990860.18 total 46.07 last 538.09) - open/close 30000 (time 1736990868.82 total 54.70 last 534.65) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 59.47 seconds: 548.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9800 (time 1736990884.49 total 10.00 last 980.00) - /unlink 19534 (time 1736990894.49 total 20.00 last 973.33) - /unlink 30000 (time 1736990903.53 total 29.04 last 1157.37) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 31.24 seconds: 1043.21 ops/second Waiting for MDT destroys to complete - open/close 5710 (time 1736990951.76 total 10.00 last 570.94) - open/close 10000 (time 1736990959.52 total 17.77 last 552.31) - open/close 15578 (time 1736990969.52 total 27.77 last 557.73) - open/close 20000 (time 1736990977.71 total 35.96 last 539.95) - open/close 25226 (time 1736990987.71 total 45.96 last 522.59) - open/close 30000 (time 1736990996.68 total 54.93 last 532.40) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 59.93 seconds: 543.82 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8447 (time 1736991012.61 total 10.00 last 844.59) - /unlink 17186 (time 1736991022.61 total 20.00 last 873.85) - /unlink 20000 (time 1736991025.75 total 23.14 last 895.74) - /unlink 29130 (time 1736991035.75 total 33.14 last 912.97) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 37.00 seconds: 881.03 ops/second Waiting for MDT destroys to complete - open/close 5008 (time 1736991093.86 total 10.00 last 500.70) - open/close 10000 (time 1736991103.62 total 19.76 last 511.47) - open/close 15396 (time 1736991113.62 total 29.76 last 539.53) - open/close 20000 (time 1736991121.76 total 37.90 last 565.70) - open/close 25255 (time 1736991131.76 total 47.90 last 525.47) - open/close 30000 (time 1736991141.21 total 57.35 last 502.37) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 62.70 seconds: 519.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9770 (time 1736991157.46 total 10.00 last 976.92) - /unlink 19611 (time 1736991167.46 total 20.00 last 984.04) - /unlink 30000 (time 1736991177.29 total 29.82 last 1057.62) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 32.19 seconds: 1012.71 ops/second Waiting for MDT destroys to complete - open/close 5514 (time 1736991240.41 total 10.00 last 551.33) - open/close 10000 (time 1736991248.70 total 18.29 last 541.04) - open/close 15317 (time 1736991258.71 total 28.29 last 531.66) - open/close 20000 (time 1736991267.89 total 37.48 last 509.61) - open/close 25445 (time 1736991277.89 total 47.48 last 544.50) - open/close 30000 (time 1736991286.70 total 56.29 last 517.48) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 60.99 seconds: 534.41 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9942 (time 1736991302.32 total 10.00 last 994.12) - /unlink 20000 (time 1736991312.24 total 19.92 last 1014.14) - /unlink 29993 (time 1736991322.24 total 29.92 last 999.20) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 32.86 seconds: 991.98 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server LNET ready to unload unloading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local oleg205-server: LNET ready to unload modules unloaded. pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2 PASS 106 (696s) Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2 oleg205-server: oleg205-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: detected 4 online CPUs by sysfs oleg205-server: Force libcfs to create 2 CPU partitions oleg205-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg205-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 801 sec =========== 20:36:20 (1736991380)