== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:30:20 (1782631820) start mds service on oleg424-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg424-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 833 (time 1782631858.27 total 10.41 last 80.04) - open/close 1531 (time 1782631868.27 total 20.41 last 69.79) - open/close 2272 (time 1782631878.27 total 30.41 last 74.08) - open/close 2945 (time 1782631888.27 total 40.41 last 67.29) - open/close 3663 (time 1782631898.33 total 50.47 last 71.37) - open/close 4341 (time 1782631908.33 total 60.47 last 67.80) - open/close 5099 (time 1782631918.69 total 70.82 last 73.23) - open/close 5840 (time 1782631928.80 total 80.94 last 73.29) - open/close 6697 (time 1782631939.36 total 91.50 last 81.15) - open/close 7427 (time 1782631949.36 total 101.50 last 73.00) - open/close 8088 (time 1782631959.36 total 111.50 last 66.09) - open/close 8789 (time 1782631969.36 total 121.50 last 70.09) - open/close 9522 (time 1782631980.32 total 132.46 last 66.89) - open/close 10000 (time 1782631987.76 total 139.90 last 64.22) - open/close 10671 (time 1782631997.76 total 149.90 last 67.10) - open/close 11396 (time 1782632007.84 total 159.98 last 71.96) - open/close 12058 (time 1782632017.84 total 169.98 last 66.18) - open/close 12813 (time 1782632028.00 total 180.14 last 74.31) - open/close 13477 (time 1782632038.05 total 190.19 last 66.08) - open/close 14181 (time 1782632048.35 total 200.49 last 68.35) - open/close 14908 (time 1782632058.52 total 210.66 last 71.44) - open/close 15587 (time 1782632068.53 total 220.66 last 67.89) - open/close 16273 (time 1782632078.53 total 230.67 last 68.58) - open/close 16958 (time 1782632088.67 total 240.81 last 67.53) - open/close 17627 (time 1782632098.67 total 250.81 last 66.89) - open/close 18332 (time 1782632108.83 total 260.97 last 69.43) - open/close 18972 (time 1782632119.71 total 271.85 last 58.81) - open/close 19560 (time 1782632130.59 total 282.73 last 54.06) - open/close 20000 (time 1782632136.81 total 288.95 last 70.71) - open/close 20680 (time 1782632146.81 total 298.95 last 67.99) - open/close 21381 (time 1782632156.90 total 309.04 last 69.52) - open/close 23449 (time 1782632166.90 total 319.04 last 206.80) - open/close 27804 (time 1782632176.90 total 329.04 last 435.49) - open/close 30000 (time 1782632181.98 total 334.12 last 431.99) - open/close 34391 (time 1782632191.98 total 344.12 last 439.08) - open/close 38705 (time 1782632201.98 total 354.12 last 431.34) - open/close 40000 (time 1782632205.34 total 357.48 last 385.35) - open/close 44208 (time 1782632215.34 total 367.48 last 420.75) - open/close 48332 (time 1782632225.35 total 377.48 last 412.32) total: 50000 open/close in 381.52 seconds: 131.06 ops/second - unlinked 0 (time 1782632230 ; total 0 ; last 0) - unlinked 10000 (time 1782632247 ; total 17 ; last 17) - unlinked 20000 (time 1782632262 ; total 32 ; last 15) - unlinked 30000 (time 1782632279 ; total 49 ; last 17) - unlinked 40000 (time 1782632295 ; total 65 ; last 16) total: 50000 unlinks in 81 seconds: 617.283936 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local oleg424-server: modules unloaded. pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2