-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:35:20 EDT 2025 oleg255-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg255-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server oleg255-server: oleg255-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg255-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg255-server: oleg255-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg255-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server == conf-sanity test 121: failover MGS ==================== 09:37:50 (1761313070) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:) start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Failing mgs on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server reboot facets: mgs Failover mgs to oleg255-server mount facets: mgs Starting mgs: /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 95 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid pdsh@oleg255-client: oleg255-client: ssh exited with exit code 95 stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Failing mgs on oleg255-server Stopping /mnt/lustre-mds1 (opts:) on oleg255-server reboot facets: mgs Failover mgs to oleg255-server mount facets: mgs Starting mgs: /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 95 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid pdsh@oleg255-client: oleg255-client: ssh exited with exit code 95 stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server PASS 121 (187s) == conf-sanity test 122a: Check OST sequence update ====== 09:40:57 (1761313257) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) oleg255-server: oleg255-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre fail_loc=0 - open/close 697 (time 1761313385.40 total 10.01 last 69.64) total: 1000 open/close in 14.02 seconds: 71.35 ops/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg255-server: LNET ready to unload oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 PASS 122a (175s) == conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows ========================================================== 09:43:52 (1761313432) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre obdfilter.lustre-OST0000.precreate_batch=256 fail_loc=0 On OST0, 49667 inodes available. Want 100500. rc=0 - open/close 725 (time 1761313571.97 total 10.01 last 72.46) - open/close 1488 (time 1761313581.98 total 20.01 last 76.25) - open/close 2203 (time 1761313591.98 total 30.02 last 71.47) - open/close 2888 (time 1761313601.99 total 40.03 last 68.41) - open/close 3638 (time 1761313612.01 total 50.05 last 74.85) - open/close 4341 (time 1761313622.02 total 60.05 last 70.25) - open/close 5103 (time 1761313632.03 total 70.06 last 76.13) - open/close 5824 (time 1761313642.03 total 80.07 last 72.07) - open/close 6413 (time 1761313652.04 total 90.07 last 58.86) - open/close 7012 (time 1761313662.04 total 100.08 last 59.89) - open/close 7661 (time 1761313672.04 total 110.08 last 64.90) - open/close 8379 (time 1761313682.06 total 120.10 last 71.66) - open/close 9122 (time 1761313692.06 total 130.10 last 74.30) - open/close 9910 (time 1761313702.06 total 140.10 last 78.78) total: 10000 open/close in 141.04 seconds: 70.90 ops/second - unlinked 0 (time 1761313705 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 640 (time 1761313791.52 total 10.02 last 63.85) - open/close 1305 (time 1761313801.52 total 20.03 last 66.48) - open/close 1992 (time 1761313811.53 total 30.03 last 68.66) - open/close 2702 (time 1761313821.53 total 40.04 last 70.97) - open/close 3425 (time 1761313831.53 total 50.04 last 72.29) - open/close 4101 (time 1761313841.55 total 60.06 last 67.48) - open/close 4733 (time 1761313851.57 total 70.07 last 63.09) - open/close 5457 (time 1761313861.58 total 80.09 last 72.30) - open/close 6027 (time 1761313871.59 total 90.10 last 56.92) - open/close 6617 (time 1761313881.61 total 100.12 last 58.90) - open/close 7236 (time 1761313891.62 total 110.12 last 61.87) - open/close 7951 (time 1761313901.62 total 120.13 last 71.46) - open/close 8682 (time 1761313911.62 total 130.13 last 73.09) - open/close 9400 (time 1761313921.62 total 140.13 last 71.79) total: 10000 open/close in 148.40 seconds: 67.39 ops/second - unlinked 0 (time 1761313933 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 683 (time 1761314027.32 total 10.01 last 68.27) - open/close 1427 (time 1761314037.32 total 20.01 last 74.37) - open/close 2185 (time 1761314047.33 total 30.01 last 75.78) - open/close 2928 (time 1761314057.34 total 40.02 last 74.22) - open/close 3697 (time 1761314067.34 total 50.03 last 76.86) - open/close 4407 (time 1761314077.34 total 60.03 last 71.00) - open/close 5075 (time 1761314087.35 total 70.03 last 66.76) - open/close 5710 (time 1761314097.37 total 80.05 last 63.36) - open/close 6385 (time 1761314107.38 total 90.06 last 67.44) - open/close 7113 (time 1761314117.38 total 100.07 last 72.78) - open/close 7804 (time 1761314127.40 total 110.08 last 68.98) - open/close 8516 (time 1761314137.41 total 120.10 last 71.11) - open/close 9169 (time 1761314147.42 total 130.11 last 65.22) - open/close 9812 (time 1761314157.43 total 140.11 last 64.29) total: 10000 open/close in 142.72 seconds: 70.07 ops/second - unlinked 0 (time 1761314164 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 556 (time 1761314262.44 total 10.01 last 55.55) - open/close 1083 (time 1761314272.46 total 20.03 last 52.60) - open/close 1680 (time 1761314282.46 total 30.03 last 59.68) - open/close 2337 (time 1761314292.48 total 40.05 last 65.60) - open/close 3039 (time 1761314302.48 total 50.05 last 70.19) - open/close 3809 (time 1761314312.49 total 60.05 last 76.97) - open/close 4518 (time 1761314322.49 total 70.06 last 70.85) - open/close 5256 (time 1761314332.50 total 80.06 last 73.77) - open/close 6001 (time 1761314342.51 total 90.08 last 74.41) - open/close 6731 (time 1761314352.52 total 100.08 last 72.95) - open/close 7475 (time 1761314362.53 total 110.09 last 74.31) - open/close 8182 (time 1761314372.53 total 120.10 last 70.69) - open/close 8938 (time 1761314382.53 total 130.10 last 75.57) - open/close 9712 (time 1761314392.54 total 140.10 last 77.38) total: 10000 open/close in 143.93 seconds: 69.48 ops/second - unlinked 0 (time 1761314400 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 759 (time 1761314497.39 total 10.00 last 75.89) - open/close 1478 (time 1761314507.39 total 20.01 last 71.86) - open/close 2211 (time 1761314517.42 total 30.03 last 73.09) - open/close 2859 (time 1761314527.43 total 40.04 last 64.74) - open/close 3471 (time 1761314537.44 total 50.05 last 61.14) - open/close 4067 (time 1761314547.45 total 60.06 last 59.57) - open/close 4742 (time 1761314557.46 total 70.07 last 67.39) - open/close 5327 (time 1761314567.47 total 80.08 last 58.45) - open/close 5993 (time 1761314577.48 total 90.09 last 66.53) - open/close 6724 (time 1761314587.50 total 100.11 last 72.99) - open/close 7462 (time 1761314597.52 total 110.13 last 73.65) - open/close 8271 (time 1761314607.53 total 120.14 last 80.82) - open/close 9073 (time 1761314617.53 total 130.14 last 80.16) - open/close 9862 (time 1761314627.54 total 140.16 last 78.80) total: 10000 open/close in 142.12 seconds: 70.36 ops/second - unlinked 0 (time 1761314633 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 711 (time 1761314731.84 total 10.01 last 71.05) - open/close 1394 (time 1761314741.85 total 20.02 last 68.22) - open/close 2118 (time 1761314751.85 total 30.02 last 72.36) - open/close 2890 (time 1761314761.86 total 40.03 last 77.15) - open/close 3607 (time 1761314771.89 total 50.06 last 71.50) - open/close 4338 (time 1761314781.89 total 60.06 last 73.09) - open/close 5023 (time 1761314791.90 total 70.07 last 68.44) - open/close 5728 (time 1761314801.91 total 80.08 last 70.42) - open/close 6420 (time 1761314811.92 total 90.09 last 69.12) - open/close 7139 (time 1761314821.93 total 100.10 last 71.82) - open/close 7859 (time 1761314831.94 total 110.11 last 71.93) - open/close 8555 (time 1761314841.95 total 120.12 last 69.54) - open/close 9315 (time 1761314851.97 total 130.14 last 75.83) total: 10000 open/close in 139.28 seconds: 71.80 ops/second - unlinked 0 (time 1761314865 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 736 (time 1761314963.96 total 10.01 last 73.54) - open/close 1435 (time 1761314973.97 total 20.02 last 69.84) - open/close 2157 (time 1761314983.97 total 30.02 last 72.19) - open/close 2885 (time 1761314993.97 total 40.02 last 72.80) - open/close 3662 (time 1761315003.99 total 50.03 last 77.61) - open/close 4384 (time 1761315014.00 total 60.04 last 72.13) - open/close 5049 (time 1761315024.00 total 70.04 last 66.46) - open/close 5737 (time 1761315034.02 total 80.06 last 68.68) - open/close 6396 (time 1761315044.02 total 90.07 last 65.87) - open/close 7072 (time 1761315054.04 total 100.08 last 67.52) - open/close 7733 (time 1761315064.04 total 110.09 last 66.06) - open/close 8482 (time 1761315074.04 total 120.09 last 74.88) - open/close 9191 (time 1761315084.05 total 130.10 last 70.85) - open/close 9923 (time 1761315094.06 total 140.11 last 73.12) total: 10000 open/close in 141.14 seconds: 70.85 ops/second - unlinked 0 (time 1761315099 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 673 (time 1761315197.12 total 10.02 last 67.18) - open/close 1353 (time 1761315207.12 total 20.02 last 67.99) - open/close 1978 (time 1761315217.13 total 30.02 last 62.47) - open/close 2650 (time 1761315227.13 total 40.03 last 67.16) - open/close 3250 (time 1761315237.14 total 50.04 last 59.94) - open/close 3880 (time 1761315247.15 total 60.04 last 62.98) - open/close 4585 (time 1761315257.15 total 70.05 last 70.47) - open/close 5266 (time 1761315267.16 total 80.05 last 68.07) - open/close 5898 (time 1761315277.17 total 90.06 last 63.13) - open/close 6406 (time 1761315287.17 total 100.07 last 50.78) - open/close 6981 (time 1761315297.18 total 110.07 last 57.46) - open/close 7635 (time 1761315307.18 total 120.08 last 65.38) - open/close 8283 (time 1761315317.18 total 130.08 last 64.78) - open/close 8930 (time 1761315327.19 total 140.08 last 64.68) - open/close 9593 (time 1761315337.19 total 150.09 last 66.26) total: 10000 open/close in 157.02 seconds: 63.69 ops/second - unlinked 0 (time 1761315348 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 721 (time 1761315445.16 total 10.01 last 72.05) - open/close 1440 (time 1761315455.16 total 20.01 last 71.89) - open/close 2124 (time 1761315465.16 total 30.01 last 68.39) - open/close 2751 (time 1761315475.17 total 40.02 last 62.61) - open/close 3400 (time 1761315485.19 total 50.04 last 64.81) - open/close 4152 (time 1761315495.20 total 60.05 last 75.12) - open/close 4886 (time 1761315505.21 total 70.06 last 73.34) - open/close 5586 (time 1761315515.21 total 80.06 last 70.00) - open/close 6195 (time 1761315525.21 total 90.06 last 60.86) - open/close 6872 (time 1761315535.22 total 100.06 last 67.68) - open/close 7570 (time 1761315545.22 total 110.07 last 69.80) - open/close 8275 (time 1761315555.23 total 120.08 last 70.40) - open/close 8992 (time 1761315565.24 total 130.09 last 71.62) - open/close 9695 (time 1761315575.25 total 140.09 last 70.27) total: 10000 open/close in 144.60 seconds: 69.16 ops/second - unlinked 0 (time 1761315583 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 766 (time 1761315677.77 total 10.00 last 76.57) - open/close 1536 (time 1761315687.78 total 20.02 last 76.88) - open/close 2312 (time 1761315697.79 total 30.03 last 77.52) - open/close 3041 (time 1761315707.80 total 40.04 last 72.85) - open/close 3767 (time 1761315717.81 total 50.05 last 72.53) - open/close 4527 (time 1761315727.81 total 60.05 last 75.99) - open/close 5262 (time 1761315737.81 total 70.05 last 73.47) - open/close 5998 (time 1761315747.82 total 80.06 last 73.55) - open/close 6740 (time 1761315757.82 total 90.06 last 74.16) - open/close 7442 (time 1761315767.85 total 100.09 last 70.04) - open/close 8137 (time 1761315777.85 total 110.09 last 69.49) - open/close 8791 (time 1761315787.86 total 120.09 last 65.36) - open/close 9464 (time 1761315797.87 total 130.11 last 67.20) total: 10000 open/close in 138.20 seconds: 72.36 ops/second - unlinked 0 (time 1761315810 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 685 (time 1761315905.28 total 10.00 last 68.47) - open/close 1351 (time 1761315915.29 total 20.02 last 66.52) - open/close 2055 (time 1761315925.30 total 30.02 last 70.35) - open/close 2735 (time 1761315935.31 total 40.03 last 67.94) - open/close 3470 (time 1761315945.31 total 50.04 last 73.46) - open/close 4200 (time 1761315955.32 total 60.04 last 72.96) - open/close 4923 (time 1761315965.32 total 70.04 last 72.30) - open/close 5671 (time 1761315975.32 total 80.05 last 74.79) - open/close 6380 (time 1761315985.33 total 90.06 last 70.82) - open/close 7103 (time 1761315995.34 total 100.07 last 72.24) - open/close 7840 (time 1761316005.35 total 110.08 last 73.62) - open/close 8540 (time 1761316015.36 total 120.08 last 69.95) - open/close 9293 (time 1761316025.36 total 130.08 last 75.29) total: 10000 open/close in 139.77 seconds: 71.55 ops/second - unlinked 0 (time 1761316038 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4140 82452 5% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1504 85088 2% /mnt/lustre[MDT:1] lustre-OST0000_UUID 142216 1644 126572 2% /mnt/lustre[OST:0] filesystem_summary: 142216 1644 126572 2% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281174 s, 3.7 MB/s fail_loc=0x80000204 Failing ost1 on oleg255-server Stopping /mnt/lustre-ost1 (opts:) on oleg255-server reboot facets: ost1 Failover ost1 to oleg255-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - open/close 0 (time 1761316207.91 total 27.71 last 0.00) total: 100 open/close in 30.43 seconds: 3.29 ops/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg255-server: LNET ready to unload oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. PASS 122b (2834s) == conf-sanity test 123aa: llog_print works with FIDs and simple names ========================================================== 10:31:06 (1761316266) start mds service on oleg255-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre 1 UP mgs MGS MGS 6 - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.155@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.155@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.155@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } PASS 123aa (95s) == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:32:41 (1761316361) PASS 123ab (9s) == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:32:50 (1761316370) PASS 123ac (9s) == conf-sanity test 123ad: llog_print shows all records == 10:32:59 (1761316379) PASS 123ad (11s) == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:33:11 (1761316391) - { index: 11, event: set_param, device: general, parameter: osc.*.max_dirty_mb, value: 462 } - { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=462 } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.155@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.155@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.155@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } - { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=1024 } - { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=462 } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.155@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.155@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.202.155@tcp(0x20000c0a8ca9b), node: 192.168.202.155@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.155@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } - { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=1024 } PASS 123ae (26s) == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:33:37 (1761316417) lctl --device MGS llog_catlist ... orig_clist: lustre-client lustre-MDT0001 lustre-MDT0000 lustre-OST0000 fail_loc=0x131b fail_val=2 new_clist: lustre-MDT0001 lustre-MDT0000 lustre-OST0000 fail_loc=0 done lctl --device lustre-MDT0000 llog_catlist ... orig_clist: [0x2:0x1:0x0]:0 fail_loc=0x131b fail_val=2 new_clist: fail_loc=0 done fail_loc=0 PASS 123af (15s) == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:33:53 (1761316433) PASS 123ag (16s) == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:34:08 (1761316448) - { index: 419, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (117s) == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:36:05 (1761316565) oleg255-server: rm: cannot remove '/tmp/f123F.conf-sanity.yaml': No such file or directory pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Unmounting FS Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre opts: Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server Writeconf checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 Remounting start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Setting configuration parameters set_param: mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: osc.lustre-OST0000-osc-MDT0001.max_dirty_mb=462 set_param: mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: osc.lustre-OST0000-osc-MDT0000.max_dirty_mb=462 set_param: osc.*.max_pages_per_rpc=a set_param: jobid_name=jobid_name=jobid_name=%e.root set_param: jobid_var=TESTNAME umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg255-server: LNET ready to unload oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 PASS 123F (148s) == conf-sanity test 124: check failover after replace_nids ========================================================== 10:38:33 (1761316713) SKIP: conf-sanity test_124 needs MDT failover setup SKIP 124 (4s) == conf-sanity test 125: check l_tunedisk only tunes OSTs and their slave devices ========================================================== 10:38:37 (1761316717) Before: mgs /dev/mapper/mds1_flakey 1279 2147483647 After: mgs /dev/mapper/mds1_flakey 1279 2147483647 Before: ost1 /dev/mapper/ost1_flakey 16383 2147483647 oleg255-server: l_tunedisk: increased '/sys/devices/virtual/block/dm-2/queue/max_sectors_kb' from 16383 to 16384 After: ost1 /dev/mapper/ost1_flakey 16384 2147483647 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 PASS 125 (35s) == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:39:12 (1761316752) umount lustre on /mnt/lustre..... stop ost1 service on oleg255-server stop mds service on oleg255-server stop mds service on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. oleg255-server: oleg255-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x60d oleg255-server: oleg255-server.virtnet: executing load_modules oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 clearing fail_loc on mds1 fail_loc=0 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 126 (58s) == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:40:11 (1761316811) umount lustre on /mnt/lustre..... stop ost1 service on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg255-server: LNET ready to unload oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. start mds service on oleg255-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Stopping clients: /mnt/lustre (opts:) pdsh@oleg255-client: no remote hosts specified check osc.lustre-OST0000-osc-MDT0000.active target updated after 0 sec (got 1) check osc.lustre-OST0000-osc-MDT0001.active target updated after 0 sec (got 1) dd: error writing '/mnt/lustre/f127.conf-sanity': No space left on device 123+0 records in 122+0 records out 127926272 bytes (128 MB, 122 MiB) copied, 10.0624 s, 12.7 MB/s 122+0 records in 122+0 records out 127926272 bytes (128 MB, 122 MiB) copied, 10.8158 s, 11.8 MB/s PASS 127 (156s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:42:47 (1761316967) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (4s) == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:42:52 (1761316972) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre opts: Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Format ost1: /dev/mapper/ost1_flakey Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Address already in use oleg255-server: The target service's index is already in use. (/dev/mapper/ost1_flakey) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 98 Start of /dev/mapper/ost1_flakey on ost1 failed 98 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Address already in use oleg255-server: The target service's index is already in use. (/dev/mapper/ost1_flakey) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 98 Start of /dev/mapper/ost1_flakey on ost1 failed 98 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x22 (OST first_time ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x122 (OST first_time writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server PASS 129 (134s) == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:45:06 (1761317106) Checking servers environments Checking clients oleg255-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Starting client oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ec6c5f28800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ec6c5f28800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.155@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Writing CONFIGS/mountdata start mds service on oleg255-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 PASS 130 (114s) == conf-sanity test 131: MDT backup restore with project ID ========================================================== 10:47:01 (1761317221) oleg255-server: debugfs 1.47.1-wc2 (08-Nov-2024) Checking servers environments Checking clients oleg255-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 17 Start of /dev/mapper/mds2_flakey on mds2 failed 17 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 17 Start of /dev/mapper/ost1_flakey on ost1 failed 17 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg255-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 17 Start of /dev/mapper/ost2_flakey on ost2 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab oleg255-server@tcp:/lustre is already mounted on /mnt/lustre Starting client oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ec6c5f28800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ec6c5f28800.idle_timeout=debug disable quota as required striped dir -i1 -c2 -H crush /mnt/lustre/d131.conf-sanity total: 512 open/close in 6.99 seconds: 73.21 ops/second striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d131.conf-sanity.inherit total: 128 open/close in 1.85 seconds: 69.28 ops/second Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:) Stopping client oleg255-client.virtnet /mnt/lustre opts: Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server file-level backup/restore on mds1:/dev/mapper/mds1_flakey backup data reformat new device Format mds1: /dev/mapper/mds1_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' file-level backup/restore on mds2:/dev/mapper/mds2_flakey backup data reformat new device Format mds2: /dev/mapper/mds2_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' Checking servers environments Checking clients oleg255-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Starting client oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre Started clients oleg255-client.virtnet: 192.168.202.155@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ec6e16c5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ec6e16c5800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required PASS 131 (296s) == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:51:57 (1761317517) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg255-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server oleg255-server: oleg255-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled Writing CONFIGS/mountdata start mds service on oleg255-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server PASS 132 (167s) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) oleg255-server: oleg255-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4814 sec ========== 10:55:36 (1761317736)