== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:58:33 (1709787513) start mds service on oleg402-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg402-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg402-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock oleg402-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 147 (time 1709787568.39 total 10.77 last 13.65) - open/close 407 (time 1709787578.65 total 21.03 last 25.35) - open/close 684 (time 1709787588.66 total 31.04 last 27.68) - open/close 957 (time 1709787598.67 total 41.05 last 27.27) - open/close 1248 (time 1709787608.77 total 51.15 last 28.81) - open/close 1534 (time 1709787619.57 total 61.95 last 26.47) - open/close 1694 (time 1709787629.62 total 72.00 last 15.92) - open/close 1932 (time 1709787640.59 total 82.96 last 21.71) - open/close 2242 (time 1709787650.59 total 92.97 last 30.98) - open/close 2430 (time 1709787660.87 total 103.25 last 18.29) - open/close 2689 (time 1709787671.78 total 114.16 last 23.74) - open/close 2830 (time 1709787681.99 total 124.37 last 13.80) - open/close 3088 (time 1709787692.33 total 134.71 last 24.96) - open/close 3275 (time 1709787702.35 total 144.73 last 18.67) - open/close 3401 (time 1709787712.36 total 154.74 last 12.58) - open/close 3580 (time 1709787722.37 total 164.74 last 17.90) - open/close 3762 (time 1709787732.80 total 175.18 last 17.43) - open/close 3925 (time 1709787743.77 total 186.15 last 14.87) - open/close 4109 (time 1709787754.16 total 196.54 last 17.71) - open/close 4314 (time 1709787764.16 total 206.54 last 20.49) - open/close 4449 (time 1709787774.33 total 216.71 last 13.28) - open/close 4660 (time 1709787785.17 total 227.55 last 19.46) - open/close 4860 (time 1709787795.22 total 237.60 last 19.91) - open/close 5053 (time 1709787805.35 total 247.73 last 19.03) - open/close 5256 (time 1709787815.43 total 257.81 last 20.16) - open/close 5423 (time 1709787826.15 total 268.53 last 15.58) - open/close 5603 (time 1709787836.17 total 278.55 last 17.96) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-5713) error: No space left on device total: 5713 open/close in 284.07 seconds: 20.11 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8267: kill: (3200) - No such process - unlinked 0 (time 1709787845 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-5713) error: No such file or directory total: 5713 unlinks in 33 seconds: 173.121216 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server unloading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2