== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:59:59 (1780088399) start mds service on oleg160-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: MODOPTS_LIBCFS= oleg160-server: Force libcfs to create 2 CPU partitions oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg160-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg160-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 585 (time 1780088457.11 total 10.62 last 55.06) - open/close 1115 (time 1780088467.61 total 21.12 last 50.49) - open/close 1752 (time 1780088478.23 total 31.75 last 59.96) - open/close 2434 (time 1780088488.98 total 42.50 last 63.44) - open/close 2847 (time 1780088499.16 total 52.67 last 40.58) - open/close 3347 (time 1780088509.47 total 62.98 last 48.53) - open/close 4088 (time 1780088519.47 total 72.98 last 74.10) - open/close 4858 (time 1780088529.56 total 83.07 last 76.26) - open/close 5608 (time 1780088539.57 total 93.08 last 74.98) - open/close 6408 (time 1780088549.72 total 103.23 last 78.77) - open/close 7289 (time 1780088560.09 total 113.60 last 84.97) - open/close 8086 (time 1780088570.14 total 123.65 last 79.30) - open/close 8899 (time 1780088580.32 total 133.83 last 79.88) - open/close 9581 (time 1780088590.32 total 143.83 last 68.20) - open/close 10000 (time 1780088597.42 total 150.93 last 59.00) - open/close 10800 (time 1780088608.41 total 161.92 last 72.80) - open/close 11559 (time 1780088618.41 total 171.92 last 75.89) - open/close 12073 (time 1780088628.83 total 182.34 last 49.34) - open/close 12666 (time 1780088639.58 total 193.09 last 55.16) - open/close 13412 (time 1780088649.58 total 203.09 last 74.60) - open/close 14206 (time 1780088659.58 total 213.09 last 79.39) - open/close 14860 (time 1780088670.43 total 223.94 last 60.29) - open/close 15663 (time 1780088680.43 total 233.94 last 80.30) - open/close 16427 (time 1780088690.43 total 243.94 last 76.37) - open/close 17060 (time 1780088701.15 total 254.66 last 59.08) - open/close 17505 (time 1780088711.32 total 264.84 last 43.72) - open/close 18118 (time 1780088721.82 total 275.33 last 58.42) - open/close 18843 (time 1780088732.70 total 286.21 last 66.64) - open/close 19348 (time 1780088743.00 total 296.52 last 48.99) - open/close 20000 (time 1780088749.42 total 302.94 last 101.55) - open/close 24846 (time 1780088759.43 total 312.94 last 484.54) - open/close 30000 (time 1780088769.00 total 322.52 last 538.08) - open/close 34914 (time 1780088779.00 total 332.52 last 491.38) - open/close 39678 (time 1780088789.01 total 342.52 last 476.36) - open/close 44541 (time 1780088799.01 total 352.52 last 486.29) - open/close 49246 (time 1780088809.01 total 362.52 last 470.45) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 363.39 seconds: 136.58 ops/second - unlinked 0 (time 1780088811 ; total 0 ; last 0) - unlinked 10000 (time 1780088823 ; total 12 ; last 12) - unlinked 20000 (time 1780088835 ; total 24 ; last 12) - unlinked 30000 (time 1780088847 ; total 36 ; last 12) - unlinked 40000 (time 1780088858 ; total 47 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 57 seconds: 870.719299 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2