-----============= acceptance-small: conf-sanity ============----- Wed Jan 8 12:35:59 EST 2025 excepting tests: 32newtarball 110 Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg414-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg414-server oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg414-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg414-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg414-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:37:24 (1736357844) Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg414-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5711 (time 1736357885.08 total 10.00 last 571.07) - open/close 10000 (time 1736357892.60 total 17.52 last 570.48) - open/close 15665 (time 1736357902.60 total 27.52 last 566.46) - open/close 20000 (time 1736357910.16 total 35.08 last 573.51) - open/close 25595 (time 1736357920.16 total 45.08 last 559.49) - open/close 30000 (time 1736357927.59 total 52.50 last 593.40) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 56.97 seconds: 572.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1736357942.07 total 9.18 last 1088.77) - /unlink 20000 (time 1736357950.87 total 17.99 last 1136.22) - /unlink 30000 (time 1736357959.99 total 27.10 last 1096.88) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 29.40 seconds: 1108.57 ops/second Waiting for MDT destroys to complete - open/close 5876 (time 1736358008.58 total 10.00 last 587.56) - open/close 10000 (time 1736358016.21 total 17.63 last 540.68) - open/close 15389 (time 1736358026.21 total 27.63 last 538.84) - open/close 20000 (time 1736358033.76 total 35.17 last 611.26) - open/close 26314 (time 1736358043.76 total 45.17 last 631.29) - open/close 30000 (time 1736358049.64 total 51.05 last 626.81) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 55.46 seconds: 587.67 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1736358063.30 total 8.42 last 1186.99) - /unlink 20000 (time 1736358072.20 total 17.32 last 1123.77) - /unlink 30000 (time 1736358080.75 total 25.88 last 1168.66) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 28.04 seconds: 1162.59 ops/second Waiting for MDT destroys to complete - open/close 5954 (time 1736358125.21 total 10.00 last 595.32) - open/close 10000 (time 1736358132.30 total 17.10 last 570.04) - open/close 16382 (time 1736358142.30 total 27.10 last 638.20) - open/close 20000 (time 1736358147.83 total 32.62 last 654.98) - open/close 26869 (time 1736358157.83 total 42.62 last 686.89) - open/close 30000 (time 1736358162.37 total 47.17 last 688.54) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 51.27 seconds: 635.73 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1736358175.65 total 8.34 last 1199.15) - /unlink 20000 (time 1736358183.66 total 16.35 last 1248.88) - /unlink 30000 (time 1736358191.54 total 24.23 last 1269.19) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 26.24 seconds: 1242.10 ops/second Waiting for MDT destroys to complete - open/close 5462 (time 1736358242.63 total 10.00 last 546.15) - open/close 10000 (time 1736358250.88 total 18.25 last 550.36) - open/close 15586 (time 1736358260.88 total 28.25 last 558.60) - open/close 20000 (time 1736358268.41 total 35.77 last 586.33) - open/close 25822 (time 1736358278.41 total 45.77 last 582.20) - open/close 30000 (time 1736358285.41 total 52.78 last 596.49) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 57.15 seconds: 570.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1736358299.95 total 9.29 last 1076.81) - /unlink 20000 (time 1736358309.53 total 18.87 last 1043.74) - /unlink 30000 (time 1736358318.55 total 27.89 last 1108.87) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 30.19 seconds: 1079.75 ops/second Waiting for MDT destroys to complete - open/close 6021 (time 1736358357.88 total 10.00 last 602.10) - open/close 10000 (time 1736358364.39 total 16.51 last 611.44) - open/close 16054 (time 1736358374.39 total 26.51 last 605.39) - open/close 20000 (time 1736358381.04 total 33.16 last 593.03) - open/close 25723 (time 1736358391.04 total 43.16 last 572.21) - open/close 30000 (time 1736358398.50 total 50.62 last 573.79) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 55.04 seconds: 592.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1736358412.78 total 9.02 last 1108.36) - /unlink 20000 (time 1736358421.73 total 17.97 last 1118.08) - /unlink 30000 (time 1736358431.04 total 27.28 last 1073.37) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 29.77 seconds: 1094.74 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server LNET ready to unload unloading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local oleg414-server: LNET ready to unload modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 PASS 106 (640s) Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 743 sec =========== 12:48:23 (1736358503)