== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:47:13 (1784126833) start mds service on oleg339-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg339-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg339-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 685 (time 1784126887.21 total 10.03 last 68.30) - open/close 1293 (time 1784126897.21 total 20.03 last 60.79) - open/close 1918 (time 1784126907.21 total 30.03 last 62.49) - open/close 2434 (time 1784126920.11 total 42.93 last 40.01) - open/close 3091 (time 1784126930.28 total 53.10 last 64.59) - open/close 3734 (time 1784126941.22 total 64.05 last 58.74) - open/close 4437 (time 1784126951.66 total 74.48 last 67.37) - open/close 5111 (time 1784126962.03 total 84.85 last 65.01) - open/close 5717 (time 1784126972.07 total 94.90 last 60.32) - open/close 6391 (time 1784126982.44 total 105.27 last 64.99) - open/close 7050 (time 1784126992.68 total 115.50 last 64.39) - open/close 7734 (time 1784127002.86 total 125.68 last 67.21) - open/close 8289 (time 1784127013.80 total 136.63 last 50.70) - open/close 9017 (time 1784127023.80 total 146.63 last 72.79) - open/close 9633 (time 1784127033.83 total 156.66 last 61.42) - open/close 10000 (time 1784127038.00 total 160.82 last 88.12) - open/close 10538 (time 1784127048.81 total 171.63 last 49.77) - open/close 11144 (time 1784127058.99 total 181.81 last 59.54) - open/close 11857 (time 1784127069.48 total 192.30 last 67.96) - open/close 12558 (time 1784127079.97 total 202.80 last 66.79) - open/close 13184 (time 1784127090.60 total 213.43 last 58.90) - open/close 13624 (time 1784127100.61 total 223.43 last 43.96) - open/close 14135 (time 1784127110.76 total 233.58 last 50.35) - open/close 14646 (time 1784127120.81 total 243.63 last 50.86) - open/close 15010 (time 1784127131.18 total 254.00 last 35.11) - open/close 15585 (time 1784127141.18 total 264.00 last 57.48) - open/close 16196 (time 1784127151.40 total 274.23 last 59.76) - open/close 16835 (time 1784127161.58 total 284.40 last 62.80) - open/close 17482 (time 1784127171.69 total 294.51 last 64.00) - open/close 18152 (time 1784127181.86 total 304.69 last 65.84) - open/close 18835 (time 1784127192.23 total 315.06 last 65.87) - open/close 20000 (time 1784127201.47 total 324.29 last 126.12) - open/close 24019 (time 1784127211.47 total 334.29 last 401.86) - open/close 28124 (time 1784127221.47 total 344.29 last 410.49) - open/close 30000 (time 1784127226.87 total 349.70 last 347.16) - open/close 33704 (time 1784127236.87 total 359.70 last 370.38) - open/close 36888 (time 1784127246.88 total 369.70 last 318.31) - open/close 40000 (time 1784127256.73 total 379.55 last 315.87) - open/close 43745 (time 1784127266.73 total 389.56 last 374.42) - open/close 47599 (time 1784127276.73 total 399.56 last 385.34) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 403.94 seconds: 122.87 ops/second - unlinked 0 (time 1784127282 ; total 0 ; last 0) - unlinked 10000 (time 1784127294 ; total 12 ; last 12) - unlinked 20000 (time 1784127307 ; total 25 ; last 13) - unlinked 30000 (time 1784127320 ; total 38 ; last 13) - unlinked 40000 (time 1784127334 ; total 52 ; last 14) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 66 seconds: 751.984863 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2