== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:46:41 (1783518401) start mds service on oleg157-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing load_modules_local oleg157-server: Loading modules from /home/green/git/lustre-release/lustre oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg157-server: detected 4 online CPUs by sysfs oleg157-server: MODOPTS_LIBCFS= oleg157-server: Force libcfs to create 2 CPU partitions oleg157-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg157-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg157-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 461 (time 1783518440.28 total 10.92 last 42.21) - open/close 1060 (time 1783518450.45 total 21.09 last 58.88) - open/close 1691 (time 1783518461.01 total 31.65 last 59.76) - open/close 2256 (time 1783518471.01 total 41.65 last 56.50) - open/close 2822 (time 1783518482.07 total 52.71 last 51.18) - open/close 3484 (time 1783518494.61 total 65.26 last 52.77) - open/close 3949 (time 1783518505.49 total 76.14 last 42.75) - open/close 4341 (time 1783518516.50 total 87.14 last 35.61) - open/close 5009 (time 1783518526.93 total 97.58 last 64.03) - open/close 5495 (time 1783518536.93 total 107.58 last 48.60) - open/close 6127 (time 1783518549.40 total 120.04 last 50.69) - open/close 6528 (time 1783518561.36 total 132.01 last 33.52) - open/close 7156 (time 1783518572.05 total 142.69 last 58.77) - open/close 7743 (time 1783518582.48 total 153.13 last 56.27) - open/close 8181 (time 1783518595.54 total 166.19 last 33.54) - open/close 8629 (time 1783518605.54 total 176.19 last 44.80) - open/close 9017 (time 1783518616.40 total 187.05 last 35.72) - open/close 9522 (time 1783518626.41 total 197.05 last 50.48) - open/close 9978 (time 1783518636.43 total 207.08 last 45.47) - open/close 10686 (time 1783518646.44 total 217.08 last 70.77) - open/close 11245 (time 1783518657.10 total 227.75 last 52.40) - open/close 11741 (time 1783518667.54 total 238.19 last 47.53) - open/close 12194 (time 1783518677.84 total 248.49 last 43.98) - open/close 12811 (time 1783518687.84 total 258.49 last 61.69) - open/close 13428 (time 1783518697.84 total 268.49 last 61.70) - open/close 13952 (time 1783518708.56 total 279.21 last 48.88) - open/close 14569 (time 1783518719.32 total 289.96 last 57.39) - open/close 15226 (time 1783518730.13 total 300.77 last 60.75) - open/close 15894 (time 1783518741.07 total 311.72 last 61.03) - open/close 16730 (time 1783518751.08 total 321.72 last 83.59) - open/close 20000 (time 1783518758.09 total 328.74 last 466.20) - open/close 24672 (time 1783518768.09 total 338.74 last 467.10) - open/close 29560 (time 1783518778.09 total 348.74 last 488.76) - open/close 34219 (time 1783518788.09 total 358.74 last 465.87) - open/close 38794 (time 1783518798.09 total 368.74 last 457.43) - open/close 40000 (time 1783518800.92 total 371.57 last 426.51) - open/close 44789 (time 1783518810.92 total 381.57 last 478.89) - open/close 49611 (time 1783518820.92 total 391.57 last 482.19) total: 50000 open/close in 392.36 seconds: 127.43 ops/second - unlinked 0 (time 1783518823 ; total 0 ; last 0) - unlinked 10000 (time 1783518839 ; total 16 ; last 16) - unlinked 20000 (time 1783518853 ; total 30 ; last 14) - unlinked 30000 (time 1783518868 ; total 45 ; last 15) - unlinked 40000 (time 1783518884 ; total 61 ; last 16) total: 50000 unlinks in 77 seconds: 649.350647 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server unloading modules via unload_modules_local on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local oleg157-server: modules unloaded. pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2