== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:59:40 (1789189180) start mds service on oleg158-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 11 13:50 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: kmemleak disabled oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg158-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg158-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 707 (time 1789189233.91 total 10.00 last 70.69) - open/close 1326 (time 1789189244.55 total 20.64 last 58.20) - open/close 1909 (time 1789189255.17 total 31.26 last 54.88) - open/close 2651 (time 1789189266.05 total 42.14 last 68.19) - open/close 3254 (time 1789189276.62 total 52.70 last 57.08) - open/close 3763 (time 1789189286.98 total 63.07 last 49.12) - open/close 4324 (time 1789189297.48 total 73.56 last 53.44) - open/close 4982 (time 1789189308.16 total 84.25 last 61.58) - open/close 5624 (time 1789189319.11 total 95.20 last 58.65) - open/close 6067 (time 1789189329.80 total 105.89 last 41.44) - open/close 6674 (time 1789189339.97 total 116.06 last 59.67) - open/close 7350 (time 1789189350.28 total 126.36 last 65.61) - open/close 8074 (time 1789189360.32 total 136.41 last 72.06) - open/close 8745 (time 1789189370.32 total 146.41 last 67.09) - open/close 9388 (time 1789189380.33 total 156.41 last 64.29) - open/close 9932 (time 1789189390.33 total 166.41 last 54.39) - open/close 10609 (time 1789189400.33 total 176.42 last 67.69) - open/close 11310 (time 1789189410.33 total 186.42 last 70.09) - open/close 11810 (time 1789189421.13 total 197.21 last 46.31) - open/close 12474 (time 1789189431.24 total 207.33 last 65.65) - open/close 13178 (time 1789189441.73 total 217.82 last 67.09) - open/close 13827 (time 1789189451.78 total 227.87 last 64.58) - open/close 14596 (time 1789189461.79 total 237.87 last 76.88) - open/close 14890 (time 1789189474.31 total 250.40 last 23.48) - open/close 15482 (time 1789189484.31 total 260.40 last 59.19) - open/close 16128 (time 1789189494.47 total 270.56 last 63.60) - open/close 16925 (time 1789189505.22 total 281.31 last 74.14) - open/close 17516 (time 1789189515.85 total 291.93 last 55.61) - open/close 18272 (time 1789189525.85 total 301.93 last 75.59) - open/close 18882 (time 1789189536.71 total 312.80 last 56.15) - open/close 20000 (time 1789189538.99 total 315.08 last 489.53) - open/close 24391 (time 1789189548.99 total 325.08 last 439.09) - open/close 28954 (time 1789189559.00 total 335.08 last 456.30) - open/close 30000 (time 1789189561.05 total 337.14 last 509.30) - open/close 33954 (time 1789189571.06 total 347.15 last 394.91) - open/close 37976 (time 1789189581.06 total 357.15 last 402.07) - open/close 40000 (time 1789189587.04 total 363.13 last 338.72) - open/close 44096 (time 1789189597.04 total 373.13 last 409.60) - open/close 48164 (time 1789189607.04 total 383.13 last 406.76) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 386.43 seconds: 128.43 ops/second - unlinked 0 (time 1789189611 ; total 0 ; last 0) - unlinked 10000 (time 1789189623 ; total 12 ; last 12) - unlinked 20000 (time 1789189635 ; total 24 ; last 12) - unlinked 30000 (time 1789189649 ; total 38 ; last 14) - unlinked 40000 (time 1789189663 ; total 52 ; last 14) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 66 seconds: 751.984863 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: kmemleak disabled modules unloaded. pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2