== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:53:31 (1709787211) start mds service on oleg312-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg312-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg312-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg312-client.virtnet: -o user_xattr,flock oleg312-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 198 (time 1709787269.57 total 10.81 last 18.32) - open/close 439 (time 1709787279.58 total 20.82 last 24.07) - open/close 588 (time 1709787289.61 total 30.85 last 14.86) - open/close 788 (time 1709787300.23 total 41.46 last 18.84) - open/close 1075 (time 1709787311.13 total 52.37 last 26.32) - open/close 1251 (time 1709787321.14 total 62.38 last 17.58) - open/close 1381 (time 1709787332.14 total 73.37 last 11.82) - open/close 1624 (time 1709787342.45 total 83.69 last 23.56) - open/close 1780 (time 1709787352.92 total 94.16 last 14.90) - open/close 1962 (time 1709787363.88 total 105.12 last 16.60) - open/close 2115 (time 1709787374.17 total 115.40 last 14.88) - open/close 2344 (time 1709787384.77 total 126.00 last 21.60) - open/close 2523 (time 1709787395.26 total 136.49 last 17.06) - open/close 2716 (time 1709787405.72 total 146.95 last 18.45) - open/close 2867 (time 1709787415.74 total 156.98 last 15.06) - open/close 3104 (time 1709787425.77 total 167.01 last 23.64) - open/close 3195 (time 1709787435.77 total 177.01 last 9.10) - open/close 3334 (time 1709787446.51 total 187.74 last 12.95) - open/close 3573 (time 1709787456.52 total 197.75 last 23.88) - open/close 3816 (time 1709787466.85 total 208.08 last 23.52) - open/close 4038 (time 1709787477.61 total 218.85 last 20.63) - open/close 4187 (time 1709787487.82 total 229.05 last 14.60) - open/close 4337 (time 1709787497.84 total 239.08 last 14.96) - open/close 4506 (time 1709787508.73 total 249.97 last 15.52) - open/close 4663 (time 1709787520.00 total 261.24 last 13.93) - open/close 4845 (time 1709787530.02 total 271.25 last 18.17) - open/close 5035 (time 1709787540.81 total 282.04 last 17.61) - open/close 5295 (time 1709787551.59 total 292.83 last 24.11) - open/close 5472 (time 1709787562.26 total 303.50 last 16.59) - open/close 5688 (time 1709787572.33 total 313.56 last 21.45) - open/close 5865 (time 1709787582.82 total 324.06 last 16.86) - open/close 6080 (time 1709787592.84 total 334.08 last 21.47) - open/close 6272 (time 1709787603.53 total 344.77 last 17.96) - open/close 6420 (time 1709787613.93 total 355.16 last 14.23) - open/close 6587 (time 1709787623.93 total 365.17 last 16.70) - open/close 6727 (time 1709787633.96 total 375.20 last 13.96) - open/close 6900 (time 1709787644.42 total 385.66 last 16.54) - open/close 7071 (time 1709787654.43 total 395.67 last 17.08) - open/close 7209 (time 1709787664.66 total 405.90 last 13.49) - open/close 7332 (time 1709787674.67 total 415.91 last 12.29) - open/close 7499 (time 1709787684.86 total 426.10 last 16.39) - open/close 7679 (time 1709787695.69 total 436.92 last 16.63) - open/close 7780 (time 1709787705.71 total 446.94 last 10.08) - open/close 8007 (time 1709787715.72 total 456.96 last 22.67) - open/close 8195 (time 1709787725.75 total 466.99 last 18.74) - open/close 8386 (time 1709787736.27 total 477.50 last 18.16) - open/close 8552 (time 1709787746.76 total 488.00 last 15.82) - open/close 8743 (time 1709787756.77 total 498.01 last 19.07) - open/close 8871 (time 1709787766.79 total 508.02 last 12.79) - open/close 9034 (time 1709787777.69 total 518.92 last 14.95) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9201) error: No space left on device total: 9201 open/close in 525.88 seconds: 17.50 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8267: kill: (3156) - No such process - unlinked 0 (time 1709787790 ; total 1 ; last 1) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9201) error: No such file or directory total: 9201 unlinks in 83 seconds: 110.855423 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg312-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg312-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server unloading modules on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2