-----============= acceptance-small: conf-sanity ============----- Mon Dec 2 13:26:48 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg120-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg120-server oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg120-server: oleg120-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg120-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:28:31 (1733164111) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg120-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg120-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg120-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1671 (time 1733164163.63 total 10.00 last 167.09) - open/close 3391 (time 1733164173.63 total 20.00 last 171.98) - open/close 5258 (time 1733164183.64 total 30.01 last 186.62) - open/close 7285 (time 1733164193.64 total 40.01 last 202.67) - open/close 9112 (time 1733164203.64 total 50.01 last 182.65) - open/close 10000 (time 1733164208.40 total 54.77 last 186.50) - open/close 12538 (time 1733164218.41 total 64.78 last 253.72) - open/close 15330 (time 1733164228.41 total 74.78 last 279.11) - open/close 17768 (time 1733164238.41 total 84.78 last 243.75) - open/close 20000 (time 1733164246.92 total 93.29 last 262.32) - open/close 22836 (time 1733164256.92 total 103.29 last 283.57) - open/close 25653 (time 1733164266.92 total 113.29 last 281.67) - open/close 28805 (time 1733164276.93 total 123.29 last 315.09) - open/close 30000 (time 1733164281.07 total 127.44 last 288.39) - open/close 32440 (time 1733164291.07 total 137.44 last 243.98) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 137.96 seconds: 236.01 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5556 (time 1733164302.86 total 10.00 last 555.45) - /unlink 10000 (time 1733164311.13 total 18.28 last 537.17) - /unlink 15722 (time 1733164321.13 total 28.28 last 572.10) - /unlink 20000 (time 1733164328.75 total 35.89 last 562.06) - /unlink 25322 (time 1733164338.75 total 45.89 last 532.14) - /unlink 30000 (time 1733164348.01 total 55.15 last 505.19) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 59.24 seconds: 549.62 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 - open/close 2459 (time 1733164391.21 total 10.00 last 245.88) - open/close 4743 (time 1733164401.22 total 20.00 last 228.34) - open/close 7613 (time 1733164411.22 total 30.01 last 286.95) - open/close 10000 (time 1733164418.37 total 37.16 last 333.86) - open/close 12615 (time 1733164428.37 total 47.16 last 261.45) - open/close 15168 (time 1733164438.37 total 57.16 last 255.25) - open/close 18144 (time 1733164448.37 total 67.16 last 297.53) - open/close 20000 (time 1733164455.17 total 73.95 last 273.26) - open/close 22879 (time 1733164465.17 total 83.95 last 287.89) - open/close 25646 (time 1733164475.17 total 93.96 last 276.56) - open/close 28916 (time 1733164485.17 total 103.96 last 326.96) - open/close 30000 (time 1733164488.11 total 106.89 last 369.71) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 114.37 seconds: 284.69 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6481 (time 1733164506.56 total 10.00 last 648.06) - /unlink 10000 (time 1733164512.78 total 16.22 last 565.99) - /unlink 15847 (time 1733164522.78 total 26.22 last 584.43) - /unlink 20000 (time 1733164529.87 total 33.31 last 586.15) - /unlink 25116 (time 1733164539.87 total 43.31 last 511.58) - /unlink 30000 (time 1733164547.45 total 50.89 last 644.23) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 54.14 seconds: 601.43 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 - open/close 3977 (time 1733164571.14 total 10.00 last 397.69) - open/close 8050 (time 1733164581.14 total 20.00 last 407.29) - open/close 10000 (time 1733164585.60 total 24.45 last 437.90) - open/close 14465 (time 1733164595.60 total 34.45 last 446.44) - open/close 18110 (time 1733164605.60 total 44.46 last 364.42) - open/close 20000 (time 1733164611.88 total 50.73 last 301.07) - open/close 23653 (time 1733164621.88 total 60.74 last 365.24) - open/close 27124 (time 1733164631.88 total 70.74 last 347.04) - open/close 30000 (time 1733164639.49 total 78.34 last 378.16) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 83.37 seconds: 390.56 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32752 47240 41% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33007 33007 0 100% /mnt/lustre - /unlink 9356 (time 1733164655.43 total 10.00 last 935.52) - /unlink 16562 (time 1733164665.43 total 20.00 last 720.57) - /unlink 20000 (time 1733164670.10 total 24.67 last 735.61) - /unlink 26878 (time 1733164680.10 total 34.68 last 687.79) - /unlink 30000 (time 1733164684.74 total 39.31 last 673.30) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 42.83 seconds: 760.14 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 - open/close 4322 (time 1733164735.77 total 10.00 last 432.19) - open/close 8519 (time 1733164745.77 total 20.00 last 419.65) - open/close 10000 (time 1733164749.37 total 23.60 last 411.99) - open/close 14571 (time 1733164759.37 total 33.60 last 457.09) - open/close 19566 (time 1733164769.37 total 43.60 last 499.43) - open/close 24877 (time 1733164779.37 total 53.60 last 531.03) - open/close 30000 (time 1733164789.27 total 63.49 last 517.78) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 68.36 seconds: 476.28 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1733164804.76 total 9.71 last 1030.14) - /unlink 20000 (time 1733164814.25 total 19.20 last 1053.87) - /unlink 29680 (time 1733164824.25 total 29.20 last 967.90) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.12 seconds: 1013.62 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 - open/close 5170 (time 1733164885.68 total 10.00 last 516.99) - open/close 10000 (time 1733164895.43 total 19.75 last 495.27) - open/close 15133 (time 1733164905.43 total 29.75 last 513.25) - open/close 20000 (time 1733164915.10 total 39.42 last 503.38) - open/close 25562 (time 1733164925.10 total 49.42 last 556.10) - open/close 30000 (time 1733164932.30 total 56.62 last 616.42) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 60.83 seconds: 535.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1733164946.70 total 9.31 last 1074.13) - /unlink 19762 (time 1733164956.70 total 19.31 last 976.17) - /unlink 29562 (time 1733164966.71 total 29.31 last 979.91) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.37 seconds: 1005.98 ops/second Waiting for MDT destroys to complete oleg120-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg120-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 PASS 106 (911s) Stopping clients: oleg120-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg120-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 oleg120-server: oleg120-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1033 sec ========== 13:44:02 (1733165042) === conf-sanity: start cleanup 13:44:03 (1733165043) === === conf-sanity: finish cleanup 13:44:04 (1733165044) ===