-----============= acceptance-small: conf-sanity ============----- Tue Jun 3 16:13:06 EDT 2025 oleg322-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg322-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg322-server oleg322-server: oleg322-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg322-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg322-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg322-server: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg322-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:14:14 (1748981654) Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f) oleg322-server: oleg322-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg322-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg322-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock oleg322-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 268 32628 1% /mnt/lustre[OST:0] filesystem_summary: 32900 272 32628 1% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4915 (time 1748981687.50 total 10.00 last 491.45) - open/close 9918 (time 1748981697.50 total 20.00 last 500.30) - open/close 14690 (time 1748981707.50 total 30.00 last 477.11) - open/close 19977 (time 1748981717.51 total 40.00 last 528.64) - open/close 24894 (time 1748981727.51 total 50.00 last 491.68) - open/close 29711 (time 1748981737.51 total 60.01 last 481.67) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 66.11 seconds: 494.04 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 9437 (time 1748981754.51 total 10.00 last 943.64) - /unlink 20000 (time 1748981764.21 total 19.70 last 1088.81) - /unlink 30000 (time 1748981773.03 total 28.52 last 1133.98) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 30.84 seconds: 1059.15 ops/second Waiting for MDT destroys to complete - open/close 4360 (time 1748981837.76 total 10.00 last 435.91) - open/close 8846 (time 1748981847.76 total 20.00 last 448.51) - open/close 10000 (time 1748981850.36 total 22.60 last 444.07) - open/close 14677 (time 1748981860.36 total 32.60 last 467.65) - open/close 19733 (time 1748981870.36 total 42.61 last 505.51) - open/close 23972 (time 1748981880.36 total 52.61 last 423.79) - open/close 28792 (time 1748981890.36 total 62.61 last 481.95) - open/close 30000 (time 1748981892.73 total 64.97 last 510.92) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 70.39 seconds: 463.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 10000 (time 1748981908.70 total 9.65 last 1036.74) - /unlink 20000 (time 1748981917.43 total 18.37 last 1145.90) - /unlink 30000 (time 1748981926.61 total 27.55 last 1089.38) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 29.80 seconds: 1095.91 ops/second Waiting for MDT destroys to complete - open/close 5373 (time 1748981964.03 total 10.00 last 537.23) - open/close 10000 (time 1748981972.42 total 18.39 last 551.28) - open/close 15324 (time 1748981982.42 total 28.40 last 532.32) - open/close 20000 (time 1748981991.28 total 37.25 last 528.17) - open/close 25104 (time 1748982001.28 total 47.25 last 510.33) - open/close 29970 (time 1748982011.28 total 57.25 last 486.59) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 62.32 seconds: 524.06 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 9080 (time 1748982027.23 total 10.00 last 907.94) - /unlink 17124 (time 1748982037.23 total 20.00 last 804.35) - /unlink 20000 (time 1748982040.84 total 23.62 last 795.64) - /unlink 28551 (time 1748982050.84 total 33.62 last 855.00) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 38.54 seconds: 847.47 ops/second Waiting for MDT destroys to complete - open/close 3892 (time 1748982092.48 total 10.00 last 389.14) - open/close 8017 (time 1748982102.48 total 20.00 last 412.47) - open/close 10000 (time 1748982107.01 total 24.53 last 437.74) - open/close 14193 (time 1748982117.01 total 34.53 last 419.30) - open/close 18420 (time 1748982127.01 total 44.53 last 422.69) - open/close 20000 (time 1748982130.30 total 47.83 last 479.28) - open/close 24623 (time 1748982140.30 total 57.83 last 462.21) - open/close 29184 (time 1748982150.31 total 67.83 last 456.07) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 75.98 seconds: 429.85 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 8727 (time 1748982169.34 total 10.00 last 872.67) - /unlink 17687 (time 1748982179.35 total 20.00 last 895.93) - /unlink 20000 (time 1748982181.71 total 22.37 last 977.31) - /unlink 28680 (time 1748982191.71 total 32.37 last 867.91) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 36.80 seconds: 887.55 ops/second Waiting for MDT destroys to complete - open/close 5828 (time 1748982221.28 total 10.00 last 582.77) - open/close 10000 (time 1748982227.76 total 16.48 last 643.91) - open/close 16428 (time 1748982237.76 total 26.48 last 642.75) - open/close 20000 (time 1748982244.63 total 33.35 last 520.35) - open/close 25213 (time 1748982254.63 total 43.35 last 521.26) - open/close 30000 (time 1748982263.73 total 52.45 last 525.79) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 57.26 seconds: 570.39 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 10000 (time 1748982279.32 total 9.90 last 1010.61) - /unlink 20000 (time 1748982288.49 total 19.06 last 1090.61) - /unlink 30000 (time 1748982297.80 total 28.38 last 1073.50) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 30.74 seconds: 1062.56 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server LNET ready to unload unloading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing unload_modules_local oleg322-server: LNET ready to unload modules unloaded. pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 PASS 106 (695s) Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 oleg322-server: oleg322-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: Force libcfs to create 2 CPU partitions oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 779 sec =========== 16:26:05 (1748982365)