== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:01:27 (1788865287) start mds service on oleg646-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing load_modules_local oleg646-server: kmemleak disabled oleg646-server: Loading modules from /home/green/git/lustre-release/lustre oleg646-server: detected 4 online CPUs by sysfs oleg646-server: MODOPTS_LIBCFS= oleg646-server: Force libcfs to create 2 CPU partitions oleg646-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg646-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg646-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg646-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg646-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg646-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-OST0000 oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg646-client.virtnet: -o user_xattr,flock 192.168.206.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 197 (time 1788865345.15 total 10.38 last 18.98) - open/close 330 (time 1788865355.15 total 20.38 last 13.30) - open/close 503 (time 1788865365.31 total 30.54 last 17.03) - open/close 631 (time 1788865375.49 total 40.72 last 12.58) - open/close 825 (time 1788865385.86 total 51.09 last 18.71) - open/close 1048 (time 1788865396.49 total 61.72 last 20.98) - open/close 1282 (time 1788865407.37 total 72.60 last 21.51) - open/close 1424 (time 1788865417.37 total 82.60 last 14.19) - open/close 1690 (time 1788865427.37 total 92.60 last 26.59) - open/close 2025 (time 1788865437.38 total 102.60 last 33.49) - open/close 2477 (time 1788865448.26 total 113.49 last 41.53) - open/close 2933 (time 1788865458.26 total 123.49 last 45.59) - open/close 3308 (time 1788865469.12 total 134.35 last 34.53) - open/close 3463 (time 1788865479.12 total 144.35 last 15.50) - open/close 3727 (time 1788865491.72 total 156.95 last 20.95) - open/close 4023 (time 1788865502.60 total 167.83 last 27.21) - open/close 4379 (time 1788865513.22 total 178.45 last 33.53) - open/close 4674 (time 1788865523.52 total 188.75 last 28.64) - open/close 4798 (time 1788865533.53 total 198.75 last 12.39) - open/close 4948 (time 1788865543.53 total 208.76 last 15.00) - open/close 5153 (time 1788865553.73 total 218.96 last 20.09) - open/close 5589 (time 1788865564.74 total 229.97 last 39.60) - open/close 5760 (time 1788865575.68 total 240.91 last 15.63) - open/close 6131 (time 1788865586.50 total 251.73 last 34.29) - open/close 6375 (time 1788865597.12 total 262.35 last 22.96) - open/close 6945 (time 1788865607.37 total 272.60 last 55.63) - open/close 7482 (time 1788865627.39 total 292.62 last 26.82) - open/close 7566 (time 1788865638.15 total 303.38 last 7.81) - open/close 7875 (time 1788865648.46 total 313.69 last 29.98) - open/close 8318 (time 1788865658.46 total 323.69 last 44.30) - open/close 9382 (time 1788865668.61 total 333.84 last 104.85) - open/close 9570 (time 1788865679.05 total 344.28 last 18.00) - open/close 9807 (time 1788865689.79 total 355.02 last 22.07) - open/close 9954 (time 1788865699.79 total 365.02 last 14.70) - open/close 10339 (time 1788865710.72 total 375.95 last 35.23) - open/close 12830 (time 1788865720.72 total 385.95 last 249.08) - open/close 15614 (time 1788865730.72 total 395.95 last 278.36) - open/close 18017 (time 1788865740.73 total 405.95 last 240.25) - open/close 20000 (time 1788865749.24 total 414.46 last 233.02) - open/close 22760 (time 1788865759.24 total 424.47 last 275.82) - open/close 25210 (time 1788865769.24 total 434.47 last 245.00) - open/close 27700 (time 1788865779.24 total 444.47 last 248.95) - open/close 30000 (time 1788865787.94 total 453.17 last 264.47) - open/close 32573 (time 1788865797.94 total 463.17 last 257.20) - open/close 35038 (time 1788865807.95 total 473.18 last 246.43) - open/close 37818 (time 1788865817.95 total 483.18 last 277.96) - open/close 40000 (time 1788865826.25 total 491.48 last 262.73) - open/close 42259 (time 1788865836.26 total 501.49 last 225.82) - open/close 45430 (time 1788865846.26 total 511.49 last 317.05) - open/close 48305 (time 1788865856.26 total 521.49 last 287.37) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 527.63 seconds: 94.06 ops/second - unlinked 0 (time 1788865864 ; total 0 ; last 0) - unlinked 10000 (time 1788865890 ; total 26 ; last 26) - unlinked 20000 (time 1788865920 ; total 56 ; last 30) - unlinked 30000 (time 1788865953 ; total 89 ; last 33) - unlinked 40000 (time 1788865982 ; total 118 ; last 29) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 144 seconds: 344.659729 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg646-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg646-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg646-server unloading modules via unload_modules_local on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing unload_modules_local oleg646-server: kmemleak disabled modules unloaded. pdsh@oleg646-client: oleg646-client: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-server: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-client: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-server: ssh exited with exit code 2