== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:04:23 (1787101463) start mds service on oleg203-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg203-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 130 (time 1787101518.07 total 10.61 last 12.25) - open/close 362 (time 1787101528.81 total 21.36 last 21.59) - open/close 515 (time 1787101539.88 total 32.43 last 13.82) - open/close 631 (time 1787101550.76 total 43.30 last 10.67) - open/close 708 (time 1787101561.05 total 53.60 last 7.48) - open/close 844 (time 1787101572.00 total 64.54 last 12.43) - open/close 954 (time 1787101582.74 total 75.29 last 10.24) - open/close 1119 (time 1787101593.94 total 86.49 last 14.73) - open/close 1316 (time 1787101604.38 total 96.93 last 18.87) - open/close 1390 (time 1787101614.55 total 107.10 last 7.27) - open/close 1550 (time 1787101624.56 total 117.11 last 15.99) - open/close 1681 (time 1787101635.55 total 128.10 last 11.92) - open/close 1845 (time 1787101645.85 total 138.40 last 15.92) - open/close 2006 (time 1787101656.23 total 148.77 last 15.52) - open/close 2134 (time 1787101666.23 total 158.78 last 12.79) - open/close 2338 (time 1787101677.15 total 169.70 last 18.68) - open/close 2437 (time 1787101687.71 total 180.25 last 9.38) - open/close 2682 (time 1787101698.40 total 190.95 last 22.91) - open/close 2892 (time 1787101708.77 total 201.32 last 20.25) - open/close 3131 (time 1787101719.46 total 212.01 last 22.36) - open/close 3285 (time 1787101729.57 total 222.11 last 15.24) - open/close 3518 (time 1787101739.57 total 232.12 last 23.28) - open/close 3640 (time 1787101750.30 total 242.85 last 11.37) - open/close 3817 (time 1787101760.31 total 252.85 last 17.69) - open/close 3992 (time 1787101771.04 total 263.58 last 16.31) - open/close 4165 (time 1787101784.03 total 276.57 last 13.32) - open/close 4409 (time 1787101794.52 total 287.07 last 23.25) - open/close 4683 (time 1787101805.03 total 297.58 last 26.07) - open/close 4800 (time 1787101815.52 total 308.07 last 11.15) - open/close 4944 (time 1787101825.53 total 318.07 last 14.39) - open/close 5096 (time 1787101835.81 total 328.35 last 14.78) - open/close 5203 (time 1787101846.57 total 339.12 last 9.94) - open/close 5359 (time 1787101856.99 total 349.53 last 14.98) - open/close 5474 (time 1787101867.74 total 360.29 last 10.69) - open/close 5601 (time 1787101879.45 total 372.00 last 10.85) - open/close 5787 (time 1787101889.69 total 382.24 last 18.16) - open/close 6029 (time 1787101900.71 total 393.26 last 21.96) - open/close 6172 (time 1787101915.80 total 408.35 last 9.48) - open/close 6467 (time 1787101925.81 total 418.35 last 29.47) - open/close 6657 (time 1787101936.80 total 429.34 last 17.29) - open/close 6992 (time 1787101946.80 total 439.35 last 33.47) - open/close 7223 (time 1787101957.27 total 449.82 last 22.06) - open/close 7359 (time 1787101967.96 total 460.50 last 12.73) - open/close 7640 (time 1787101977.97 total 470.51 last 28.07) - open/close 7836 (time 1787101987.98 total 480.53 last 19.57) - open/close 8140 (time 1787101997.99 total 490.53 last 30.38) - open/close 8387 (time 1787102008.30 total 500.84 last 23.96) - open/close 8685 (time 1787102018.34 total 510.88 last 29.68) - open/close 9000 (time 1787102028.45 total 520.99 last 31.16) - open/close 9219 (time 1787102038.49 total 531.03 last 21.81) - open/close 10000 (time 1787102045.25 total 537.80 last 115.40) - open/close 11324 (time 1787102055.26 total 547.81 last 132.25) - open/close 12609 (time 1787102065.27 total 557.81 last 128.48) - open/close 13900 (time 1787102075.27 total 567.81 last 129.08) - open/close 15272 (time 1787102085.27 total 577.81 last 137.19) - open/close 16524 (time 1787102095.27 total 587.82 last 125.17) - open/close 17767 (time 1787102105.27 total 597.82 last 124.28) - open/close 19212 (time 1787102115.28 total 607.83 last 144.36) - open/close 20000 (time 1787102121.55 total 614.10 last 125.61) - open/close 21491 (time 1787102131.56 total 624.11 last 148.98) - open/close 23012 (time 1787102141.56 total 634.11 last 152.09) - open/close 24480 (time 1787102151.56 total 644.11 last 146.78) - open/close 25892 (time 1787102161.56 total 654.11 last 141.20) - open/close 27337 (time 1787102171.57 total 664.12 last 144.37) - open/close 28740 (time 1787102181.58 total 674.12 last 140.25) - open/close 30000 (time 1787102190.07 total 682.61 last 148.44) - open/close 31694 (time 1787102200.07 total 692.61 last 169.39) - open/close 33448 (time 1787102210.07 total 702.62 last 175.28) - open/close 34964 (time 1787102220.08 total 712.62 last 151.54) - open/close 36881 (time 1787102230.08 total 722.63 last 191.63) - open/close 39280 (time 1787102240.09 total 732.63 last 239.76) - open/close 40000 (time 1787102244.33 total 736.87 last 169.76) - open/close 41634 (time 1787102254.33 total 746.87 last 163.40) - open/close 43403 (time 1787102264.33 total 756.87 last 176.87) - open/close 44787 (time 1787102274.33 total 766.88 last 138.35) - open/close 46557 (time 1787102284.33 total 776.88 last 176.98) - open/close 48169 (time 1787102294.34 total 786.89 last 161.08) - open/close 49725 (time 1787102304.34 total 796.89 last 155.56) total: 50000 open/close in 798.77 seconds: 62.60 ops/second - unlinked 0 (time 1787102308 ; total 0 ; last 0) - unlinked 10000 (time 1787102341 ; total 33 ; last 33) - unlinked 20000 (time 1787102371 ; total 63 ; last 30) - unlinked 30000 (time 1787102406 ; total 98 ; last 35) - unlinked 40000 (time 1787102441 ; total 133 ; last 35) total: 50000 unlinks in 160 seconds: 312.500000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2