== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:30:45 (1784939445) start mds service on oleg423-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg423-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 229 (time 1784939537.72 total 10.01 last 22.88) - open/close 406 (time 1784939547.75 total 20.03 last 17.66) - open/close 631 (time 1784939558.65 total 30.94 last 20.63) - open/close 867 (time 1784939569.20 total 41.49 last 22.37) - open/close 1101 (time 1784939579.71 total 51.99 last 22.28) - open/close 1223 (time 1784939589.89 total 62.17 last 11.99) - open/close 1294 (time 1784939600.20 total 72.48 last 6.89) - open/close 1546 (time 1784939610.74 total 83.03 last 23.89) - open/close 1742 (time 1784939620.78 total 93.07 last 19.52) - open/close 1994 (time 1784939632.20 total 104.49 last 22.07) - open/close 2152 (time 1784939642.28 total 114.57 last 15.68) - open/close 2282 (time 1784939652.61 total 124.90 last 12.59) - open/close 2431 (time 1784939663.03 total 135.32 last 14.30) - open/close 2574 (time 1784939673.38 total 145.66 last 13.82) - open/close 2728 (time 1784939684.10 total 156.39 last 14.36) - open/close 2882 (time 1784939695.05 total 167.33 last 14.07) - open/close 3090 (time 1784939706.14 total 178.43 last 18.75) - open/close 3177 (time 1784939717.57 total 189.86 last 7.61) - open/close 3416 (time 1784939728.44 total 200.73 last 22.00) - open/close 3634 (time 1784939738.45 total 210.74 last 21.78) - open/close 3900 (time 1784939748.45 total 220.74 last 26.60) - open/close 4131 (time 1784939758.97 total 231.26 last 21.95) - open/close 4255 (time 1784939768.98 total 241.27 last 12.39) - open/close 4506 (time 1784939778.99 total 251.28 last 25.08) - open/close 4686 (time 1784939789.02 total 261.30 last 17.95) - open/close 4808 (time 1784939800.06 total 272.35 last 11.04) - open/close 4938 (time 1784939812.48 total 284.77 last 10.47) - open/close 5143 (time 1784939822.49 total 294.78 last 20.48) - open/close 5341 (time 1784939833.35 total 305.64 last 18.23) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-5553) error: No space left on device total: 5553 open/close in 313.10 seconds: 17.74 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (75569) - No such process - unlinked 0 (time 1784939845 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-5553) error: No such file or directory total: 5553 unlinks in 52 seconds: 106.788460 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2