== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:57:44 (1773687464) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 626 (time 1773687505.28 total 10.70 last 58.49) - open/close 1151 (time 1773687515.78 total 21.20 last 50.01) - open/close 1573 (time 1773687526.02 total 31.44 last 41.21) - open/close 2040 (time 1773687536.32 total 41.74 last 45.32) - open/close 2583 (time 1773687546.94 total 52.37 last 51.11) - open/close 3149 (time 1773687557.57 total 62.99 last 53.28) - open/close 3734 (time 1773687568.25 total 73.68 last 54.73) - open/close 4369 (time 1773687579.07 total 84.50 last 58.70) - open/close 4963 (time 1773687589.82 total 95.25 last 55.26) - open/close 5673 (time 1773687600.70 total 106.13 last 65.25) - open/close 6360 (time 1773687610.88 total 116.31 last 67.50) - open/close 7089 (time 1773687621.19 total 126.61 last 70.75) - open/close 7776 (time 1773687631.19 total 136.61 last 68.69) - open/close 8471 (time 1773687642.18 total 147.60 last 63.24) - open/close 9116 (time 1773687653.12 total 158.55 last 58.93) - open/close 9813 (time 1773687664.13 total 169.55 last 63.33) - open/close 10000 (time 1773687666.65 total 172.08 last 73.98) - open/close 10642 (time 1773687677.71 total 183.13 last 58.07) - open/close 11315 (time 1773687687.71 total 193.14 last 67.29) - open/close 11914 (time 1773687697.71 total 203.14 last 59.89) - open/close 12502 (time 1773687708.61 total 214.04 last 53.94) - open/close 13207 (time 1773687719.04 total 224.47 last 67.59) - open/close 13854 (time 1773687729.22 total 234.64 last 63.60) - open/close 14406 (time 1773687739.22 total 244.64 last 55.19) - open/close 15040 (time 1773687749.22 total 254.64 last 63.40) - open/close 15678 (time 1773687759.30 total 264.72 last 63.31) - open/close 16349 (time 1773687769.35 total 274.77 last 66.76) - open/close 16938 (time 1773687779.39 total 284.82 last 58.63) - open/close 17510 (time 1773687790.21 total 295.63 last 52.90) - open/close 18207 (time 1773687800.21 total 305.63 last 69.69) - open/close 20000 (time 1773687804.30 total 309.72 last 438.27) - open/close 24297 (time 1773687814.30 total 319.72 last 429.61) - open/close 28168 (time 1773687824.30 total 329.73 last 387.03) - open/close 30000 (time 1773687828.81 total 334.24 last 406.11) - open/close 34003 (time 1773687838.81 total 344.24 last 400.23) - open/close 38530 (time 1773687848.82 total 354.24 last 452.64) - open/close 40000 (time 1773687852.50 total 357.92 last 399.58) - open/close 44107 (time 1773687862.50 total 367.92 last 410.67) - open/close 48623 (time 1773687872.50 total 377.92 last 451.55) total: 50000 open/close in 380.84 seconds: 131.29 ops/second - unlinked 0 (time 1773687876 ; total 0 ; last 0) - unlinked 10000 (time 1773687889 ; total 13 ; last 13) - unlinked 20000 (time 1773687903 ; total 27 ; last 14) - unlinked 30000 (time 1773687916 ; total 40 ; last 13) - unlinked 40000 (time 1773687930 ; total 54 ; last 14) total: 50000 unlinks in 67 seconds: 746.268677 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2