-----============= acceptance-small: conf-sanity ============----- Fri Jan 17 11:16:19 EST 2025 excepting tests: 32newtarball 110 Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg347-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg347-server oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg347-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg347-server: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg347-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:17:46 (1737130666) Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5058 (time 1737130707.86 total 10.00 last 505.76) - open/close 10000 (time 1737130717.11 total 19.25 last 534.45) - open/close 14644 (time 1737130727.11 total 29.25 last 464.39) - open/close 19035 (time 1737130737.11 total 39.25 last 438.94) - open/close 20000 (time 1737130739.14 total 41.28 last 476.06) - open/close 25092 (time 1737130749.14 total 51.28 last 509.14) - open/close 28900 (time 1737130759.14 total 61.28 last 380.79) - open/close 30000 (time 1737130761.48 total 63.62 last 470.12) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 68.34 seconds: 476.96 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8563 (time 1737130777.09 total 10.00 last 856.22) - /unlink 15753 (time 1737130787.09 total 20.00 last 718.93) - /unlink 20000 (time 1737130793.84 total 26.75 last 629.08) - /unlink 29086 (time 1737130803.84 total 36.75 last 908.52) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 40.98 seconds: 795.36 ops/second Waiting for MDT destroys to complete - open/close 4029 (time 1737130841.17 total 10.00 last 402.90) - open/close 8486 (time 1737130851.18 total 20.00 last 445.62) - open/close 10000 (time 1737130855.20 total 24.03 last 376.06) - open/close 13660 (time 1737130865.20 total 34.03 last 366.00) - open/close 17080 (time 1737130875.20 total 44.03 last 341.97) - open/close 20000 (time 1737130883.07 total 51.90 last 371.13) - open/close 24080 (time 1737130893.07 total 61.90 last 407.96) - open/close 28242 (time 1737130903.07 total 71.90 last 416.19) - open/close 30000 (time 1737130908.03 total 76.85 last 354.87) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 83.73 seconds: 389.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6787 (time 1737130925.88 total 10.00 last 678.66) - /unlink 10000 (time 1737130930.63 total 14.76 last 675.71) - /unlink 16449 (time 1737130940.64 total 24.76 last 644.83) - /unlink 20000 (time 1737130946.46 total 30.58 last 610.04) - /unlink 27131 (time 1737130956.46 total 40.58 last 713.00) - /unlink 30000 (time 1737130960.60 total 44.72 last 692.34) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 48.06 seconds: 678.24 ops/second Waiting for MDT destroys to complete - open/close 3580 (time 1737130993.01 total 10.00 last 357.96) - open/close 7756 (time 1737131003.01 total 20.00 last 417.60) - open/close 10000 (time 1737131007.01 total 24.00 last 561.57) - open/close 15042 (time 1737131017.01 total 34.00 last 504.19) - open/close 19468 (time 1737131027.01 total 44.00 last 442.60) - open/close 24373 (time 1737131037.01 total 54.00 last 490.38) - open/close 29540 (time 1737131047.01 total 64.00 last 516.69) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 71.94 seconds: 453.09 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8119 (time 1737131065.92 total 10.00 last 811.79) - /unlink 10000 (time 1737131068.06 total 12.14 last 879.44) - /unlink 18379 (time 1737131078.06 total 22.14 last 837.89) - /unlink 20000 (time 1737131080.25 total 24.33 last 739.33) - /unlink 27960 (time 1737131090.25 total 34.33 last 795.95) - /unlink 30000 (time 1737131092.35 total 36.43 last 973.99) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 39.62 seconds: 822.65 ops/second Waiting for MDT destroys to complete - open/close 4478 (time 1737131123.84 total 10.00 last 447.68) - open/close 8049 (time 1737131133.84 total 20.00 last 357.08) - open/close 10000 (time 1737131139.41 total 25.58 last 349.89) - open/close 14684 (time 1737131149.42 total 35.58 last 468.35) - open/close 19447 (time 1737131159.42 total 45.58 last 476.27) - open/close 22760 (time 1737131169.42 total 55.58 last 331.30) - open/close 26406 (time 1737131179.42 total 65.58 last 364.60) - open/close 30000 (time 1737131187.73 total 73.90 last 432.02) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 79.67 seconds: 409.11 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6989 (time 1737131204.42 total 10.00 last 698.88) - /unlink 10000 (time 1737131208.47 total 14.04 last 744.49) - /unlink 20000 (time 1737131218.02 total 23.60 last 1046.95) - /unlink 30000 (time 1737131226.12 total 31.69 last 1235.13) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 33.96 seconds: 959.80 ops/second Waiting for MDT destroys to complete - open/close 5489 (time 1737131267.16 total 10.00 last 548.77) - open/close 10000 (time 1737131275.17 total 18.02 last 562.89) - open/close 15235 (time 1737131285.17 total 28.02 last 523.43) - open/close 20000 (time 1737131293.59 total 36.44 last 566.00) - open/close 26301 (time 1737131303.59 total 46.44 last 630.07) - open/close 30000 (time 1737131310.62 total 53.47 last 525.85) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 57.47 seconds: 567.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9128 (time 1737131325.44 total 10.00 last 912.71) - /unlink 18017 (time 1737131335.44 total 20.00 last 888.83) - /unlink 27507 (time 1737131345.44 total 30.00 last 948.97) - /unlink 30000 (time 1737131348.03 total 32.59 last 962.51) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 35.78 seconds: 911.07 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server LNET ready to unload unloading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local oleg347-server: LNET ready to unload modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 PASS 106 (732s) Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 838 sec =========== 11:30:18 (1737131418)