== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:59:00 (1709787540) start mds service on oleg343-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg343-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg343-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 202 (time 1709787593.71 total 10.69 last 18.89) - open/close 432 (time 1709787603.72 total 20.71 last 22.96) - open/close 585 (time 1709787614.17 total 31.15 last 14.65) - open/close 782 (time 1709787624.17 total 41.15 last 19.70) - open/close 1040 (time 1709787634.22 total 51.20 last 25.68) - open/close 1234 (time 1709787645.33 total 62.31 last 17.46) - open/close 1474 (time 1709787655.74 total 72.73 last 23.05) - open/close 1612 (time 1709787665.80 total 82.79 last 13.72) - open/close 1843 (time 1709787675.81 total 92.79 last 23.09) - open/close 2014 (time 1709787685.82 total 102.80 last 17.08) - open/close 2263 (time 1709787695.86 total 112.84 last 24.80) - open/close 2449 (time 1709787705.99 total 122.97 last 18.36) - open/close 2661 (time 1709787716.35 total 133.33 last 20.46) - open/close 2792 (time 1709787726.61 total 143.59 last 12.77) - open/close 2946 (time 1709787737.14 total 154.13 last 14.61) - open/close 3230 (time 1709787748.07 total 165.06 last 25.98) - open/close 3413 (time 1709787758.70 total 175.69 last 17.22) - open/close 3561 (time 1709787769.05 total 186.04 last 14.30) - open/close 3773 (time 1709787779.06 total 196.04 last 21.18) - open/close 3915 (time 1709787791.18 total 208.16 last 11.72) - open/close 4149 (time 1709787802.01 total 219.00 last 21.59) - open/close 4292 (time 1709787812.13 total 229.11 last 14.14) - open/close 4490 (time 1709787822.20 total 239.18 last 19.67) - open/close 4651 (time 1709787832.55 total 249.53 last 15.55) - open/close 4794 (time 1709787843.50 total 260.48 last 13.06) - open/close 5023 (time 1709787853.79 total 270.77 last 22.26) - open/close 5193 (time 1709787864.38 total 281.37 last 16.04) - open/close 5296 (time 1709787874.56 total 291.55 last 10.12) - open/close 5515 (time 1709787885.19 total 302.17 last 20.60) - open/close 5661 (time 1709787895.33 total 312.32 last 14.40) - open/close 5732 (time 1709787905.34 total 322.33 last 7.09) - open/close 6008 (time 1709787915.35 total 332.33 last 27.58) - open/close 6214 (time 1709787926.21 total 343.20 last 18.96) - open/close 6471 (time 1709787936.26 total 353.24 last 25.59) - open/close 6629 (time 1709787946.94 total 363.93 last 14.79) - open/close 6888 (time 1709787956.94 total 373.93 last 25.90) - open/close 7071 (time 1709787966.95 total 383.93 last 18.30) - open/close 7234 (time 1709787977.11 total 394.09 last 16.04) - open/close 7497 (time 1709787987.98 total 404.96 last 24.19) - open/close 7709 (time 1709787998.89 total 415.87 last 19.44) - open/close 7909 (time 1709788008.89 total 425.87 last 20.00) - open/close 8208 (time 1709788018.90 total 435.88 last 29.86) - open/close 8400 (time 1709788028.98 total 445.96 last 19.05) - open/close 8594 (time 1709788039.04 total 456.03 last 19.27) - open/close 8715 (time 1709788051.18 total 468.16 last 9.97) - open/close 8835 (time 1709788061.19 total 478.17 last 11.99) - open/close 8956 (time 1709788072.12 total 489.11 last 11.06) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9041) error: No space left on device total: 9041 open/close in 492.27 seconds: 18.37 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8267: kill: (3254) - No such process - unlinked 0 (time 1709788078 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9041) error: No such file or directory total: 9041 unlinks in 52 seconds: 173.865387 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server unloading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2