-----============= acceptance-small: conf-sanity ============----- Fri May 16 08:45:25 EDT 2025 oleg351-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg351-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg351-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg351-server: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg351-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg351-server: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg351-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:47:36 (1747399656) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg351-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1309 (time 1747399734.51 total 10.00 last 130.85) - open/close 2580 (time 1747399744.51 total 20.00 last 127.10) - open/close 3769 (time 1747399754.52 total 30.01 last 118.85) - open/close 4967 (time 1747399764.52 total 40.01 last 119.80) - open/close 6286 (time 1747399774.52 total 50.01 last 131.83) - open/close 7701 (time 1747399784.52 total 60.01 last 141.49) - open/close 9193 (time 1747399794.53 total 70.02 last 149.09) - open/close 10000 (time 1747399799.64 total 75.14 last 157.79) - open/close 11490 (time 1747399809.65 total 85.14 last 148.94) - open/close 12919 (time 1747399819.65 total 95.14 last 142.87) - open/close 14253 (time 1747399829.65 total 105.15 last 133.35) - open/close 15572 (time 1747399839.66 total 115.15 last 131.84) - open/close 16779 (time 1747399849.66 total 125.15 last 120.68) - open/close 18082 (time 1747399859.67 total 135.17 last 130.14) - open/close 19535 (time 1747399869.67 total 145.17 last 145.30) - open/close 20000 (time 1747399872.56 total 148.05 last 161.09) - open/close 21735 (time 1747399882.56 total 158.05 last 173.47) - open/close 23372 (time 1747399892.57 total 168.06 last 163.61) - open/close 24913 (time 1747399902.57 total 178.07 last 154.00) - open/close 26812 (time 1747399912.58 total 188.07 last 189.88) - open/close 28636 (time 1747399922.58 total 198.07 last 182.36) - open/close 30000 (time 1747399930.43 total 205.92 last 173.76) - open/close 31870 (time 1747399940.43 total 215.92 last 186.98) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 220.21 seconds: 148.01 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2926 (time 1747399956.24 total 10.00 last 292.51) - /unlink 6493 (time 1747399966.24 total 20.00 last 356.69) - /unlink 9816 (time 1747399976.25 total 30.01 last 332.01) - /unlink 13008 (time 1747399986.25 total 40.01 last 319.15) - /unlink 16422 (time 1747399996.25 total 50.02 last 341.35) - /unlink 20000 (time 1747400005.54 total 59.31 last 385.05) - /unlink 22985 (time 1747400015.54 total 69.31 last 298.48) - /unlink 26033 (time 1747400025.55 total 79.31 last 304.72) - /unlink 29094 (time 1747400035.55 total 89.31 last 306.09) - /unlink 30000 (time 1747400038.16 total 91.92 last 347.43) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 101.18 seconds: 322.14 ops/second Waiting for MDT destroys to complete - open/close 1823 (time 1747400065.42 total 10.00 last 182.24) - open/close 3711 (time 1747400075.42 total 20.00 last 188.78) - open/close 5628 (time 1747400085.43 total 30.01 last 191.63) - open/close 7810 (time 1747400095.43 total 40.01 last 218.14) - open/close 9723 (time 1747400105.43 total 50.01 last 191.28) - open/close 11446 (time 1747400115.43 total 60.01 last 172.27) - open/close 13200 (time 1747400125.44 total 70.02 last 175.28) - open/close 15107 (time 1747400135.44 total 80.02 last 190.69) - open/close 16897 (time 1747400145.44 total 90.03 last 178.92) - open/close 18983 (time 1747400155.45 total 100.03 last 208.54) - open/close 20000 (time 1747400161.08 total 105.66 last 180.59) - open/close 22049 (time 1747400171.08 total 115.66 last 204.85) - open/close 24592 (time 1747400181.08 total 125.66 last 254.27) - open/close 27022 (time 1747400191.08 total 135.67 last 242.88) - open/close 29507 (time 1747400201.09 total 145.67 last 248.44) - open/close 32339 (time 1747400211.09 total 155.67 last 283.17) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 156.75 seconds: 207.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5035 (time 1747400223.25 total 10.00 last 503.41) - /unlink 10000 (time 1747400233.09 total 19.84 last 504.57) - /unlink 14252 (time 1747400243.09 total 29.84 last 425.12) - /unlink 18631 (time 1747400253.09 total 39.84 last 437.85) - /unlink 20000 (time 1747400255.92 total 42.68 last 483.45) - /unlink 24683 (time 1747400265.92 total 52.68 last 468.29) - /unlink 29066 (time 1747400275.92 total 62.68 last 438.29) - /unlink 30000 (time 1747400278.49 total 65.24 last 364.40) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 71.24 seconds: 457.51 ops/second Waiting for MDT destroys to complete - open/close 2520 (time 1747400305.61 total 10.00 last 251.88) - open/close 5369 (time 1747400315.61 total 20.01 last 284.87) - open/close 7952 (time 1747400325.61 total 30.01 last 258.25) - open/close 10000 (time 1747400333.04 total 37.43 last 275.79) - open/close 12924 (time 1747400343.04 total 47.44 last 292.31) - open/close 15751 (time 1747400353.05 total 57.44 last 282.64) - open/close 18871 (time 1747400363.05 total 67.44 last 311.98) - open/close 20000 (time 1747400366.34 total 70.73 last 342.98) - open/close 23248 (time 1747400376.34 total 80.73 last 324.78) - open/close 26789 (time 1747400386.34 total 90.73 last 354.05) - open/close 30000 (time 1747400395.63 total 100.03 last 345.53) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 107.17 seconds: 304.12 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7704 (time 1747400413.77 total 10.00 last 770.39) - /unlink 10000 (time 1747400416.66 total 12.88 last 796.34) - /unlink 17108 (time 1747400426.66 total 22.88 last 710.71) - /unlink 20000 (time 1747400430.95 total 27.18 last 673.91) - /unlink 26764 (time 1747400440.95 total 37.18 last 676.31) - /unlink 30000 (time 1747400446.12 total 42.35 last 625.80) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 45.90 seconds: 710.07 ops/second Waiting for MDT destroys to complete - open/close 4383 (time 1747400470.80 total 10.00 last 438.23) - open/close 8699 (time 1747400480.80 total 20.00 last 431.58) - open/close 10000 (time 1747400484.38 total 23.58 last 363.29) - open/close 13738 (time 1747400494.38 total 33.59 last 373.65) - open/close 17232 (time 1747400504.39 total 43.59 last 349.32) - open/close 20000 (time 1747400512.41 total 51.61 last 345.09) - open/close 23400 (time 1747400522.41 total 61.61 last 339.95) - open/close 27440 (time 1747400532.41 total 71.61 last 403.99) - open/close 30000 (time 1747400538.47 total 77.68 last 422.19) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 82.48 seconds: 395.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8573 (time 1747400554.15 total 10.00 last 857.30) - /unlink 18010 (time 1747400564.16 total 20.00 last 943.60) - /unlink 28150 (time 1747400574.16 total 30.00 last 1013.96) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 34.75 seconds: 937.84 ops/second Waiting for MDT destroys to complete - open/close 1706 (time 1747400599.18 total 10.01 last 170.51) - open/close 3771 (time 1747400609.18 total 20.01 last 206.44) - open/close 6556 (time 1747400619.18 total 30.01 last 278.40) - open/close 9392 (time 1747400629.19 total 40.01 last 283.52) - open/close 10000 (time 1747400631.44 total 42.26 last 270.23) - open/close 12297 (time 1747400641.44 total 52.27 last 229.66) - open/close 14693 (time 1747400651.44 total 62.27 last 239.55) - open/close 17092 (time 1747400661.44 total 72.27 last 239.88) - open/close 19866 (time 1747400671.44 total 82.27 last 277.39) - open/close 22674 (time 1747400681.44 total 92.27 last 280.76) - open/close 24763 (time 1747400691.45 total 102.27 last 208.87) - open/close 27152 (time 1747400701.45 total 112.27 last 238.88) - open/close 29183 (time 1747400711.45 total 122.28 last 203.00) - open/close 30000 (time 1747400715.25 total 126.08 last 215.06) - open/close 32548 (time 1747400725.25 total 136.08 last 254.71) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 136.28 seconds: 239.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5283 (time 1747400736.53 total 10.00 last 528.16) - /unlink 10000 (time 1747400744.71 total 18.18 last 576.60) - /unlink 15381 (time 1747400754.71 total 28.19 last 538.01) - /unlink 20000 (time 1747400761.74 total 35.22 last 656.69) - /unlink 27805 (time 1747400771.74 total 45.22 last 780.49) - /unlink 30000 (time 1747400774.74 total 48.22 last 731.27) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 51.81 seconds: 629.08 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server LNET ready to unload unloading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: LNET ready to unload modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 PASS 106 (1148s) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: oleg351-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1300 sec ========== 09:07:06 (1747400826)