== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:57:28 (1709787448) start mds service on oleg324-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg324-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg324-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg324-client.virtnet: -o user_xattr,flock oleg324-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 136 (time 1709787498.58 total 10.02 last 13.57) - open/close 385 (time 1709787509.28 total 20.72 last 23.29) - open/close 645 (time 1709787519.29 total 30.73 last 25.97) - open/close 875 (time 1709787529.44 total 40.88 last 22.66) - open/close 1087 (time 1709787539.69 total 51.13 last 20.68) - open/close 1206 (time 1709787549.69 total 61.13 last 11.90) - open/close 1346 (time 1709787560.32 total 71.76 last 13.18) - open/close 1543 (time 1709787570.69 total 82.13 last 18.99) - open/close 1755 (time 1709787581.45 total 92.89 last 19.69) - open/close 1936 (time 1709787591.74 total 103.18 last 17.59) - open/close 2136 (time 1709787601.77 total 113.21 last 19.95) - open/close 2362 (time 1709787611.77 total 123.21 last 22.60) - open/close 2593 (time 1709787621.77 total 133.21 last 23.09) - open/close 2711 (time 1709787631.97 total 143.41 last 11.56) - open/close 2909 (time 1709787642.90 total 154.34 last 18.13) - open/close 3098 (time 1709787653.43 total 164.87 last 17.95) - open/close 3278 (time 1709787663.53 total 174.97 last 17.82) - open/close 3478 (time 1709787673.53 total 184.97 last 19.99) - open/close 3709 (time 1709787683.56 total 195.00 last 23.04) - open/close 3895 (time 1709787694.09 total 205.53 last 17.67) - open/close 4179 (time 1709787704.09 total 215.53 last 28.39) - open/close 4350 (time 1709787714.62 total 226.06 last 16.23) - open/close 4468 (time 1709787734.06 total 245.50 last 6.07) - open/close 4635 (time 1709787744.06 total 255.50 last 16.70) - open/close 4773 (time 1709787754.08 total 265.52 last 13.78) - open/close 5014 (time 1709787764.66 total 276.10 last 22.77) - open/close 5171 (time 1709787775.15 total 286.59 last 14.97) - open/close 5327 (time 1709787785.74 total 297.18 last 14.73) - open/close 5525 (time 1709787796.14 total 307.58 last 19.03) - open/close 5694 (time 1709787806.42 total 317.86 last 16.44) - open/close 5876 (time 1709787816.74 total 328.18 last 17.63) - open/close 6109 (time 1709787826.76 total 338.20 last 23.27) - open/close 6362 (time 1709787837.06 total 348.50 last 24.55) - open/close 6538 (time 1709787847.84 total 359.28 last 16.32) - open/close 6739 (time 1709787857.89 total 369.33 last 20.00) - open/close 6933 (time 1709787868.48 total 379.92 last 18.32) - open/close 7085 (time 1709787879.01 total 390.45 last 14.44) - open/close 7230 (time 1709787889.02 total 400.46 last 14.48) - open/close 7285 (time 1709787899.28 total 410.72 last 5.36) - open/close 7418 (time 1709787909.45 total 420.89 last 13.08) - open/close 7560 (time 1709787919.47 total 430.91 last 14.18) - open/close 7770 (time 1709787929.60 total 441.04 last 20.73) - open/close 7900 (time 1709787940.16 total 451.60 last 12.31) - open/close 8102 (time 1709787950.17 total 461.61 last 20.17) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8113) error: No space left on device total: 8113 open/close in 461.82 seconds: 17.57 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8267: kill: (4841) - No such process - unlinked 0 (time 1709787955 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8113) error: No such file or directory total: 8113 unlinks in 60 seconds: 135.216660 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg324-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg324-server unloading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2