== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:13:57 (1788362037) start mds service on oleg354-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing load_modules_local oleg354-server: Loading modules from /home/green/git/lustre-release/lustre oleg354-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg354-server: detected 4 online CPUs by sysfs oleg354-server: MODOPTS_LIBCFS= oleg354-server: Force libcfs to create 2 CPU partitions oleg354-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg354-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg354-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg354-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg354-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 268 (time 1788362103.12 total 10.01 last 26.78) - open/close 593 (time 1788362113.88 total 20.76 last 30.22) - open/close 942 (time 1788362123.88 total 30.76 last 34.90) - open/close 1226 (time 1788362134.23 total 41.11 last 27.44) - open/close 1446 (time 1788362145.24 total 52.13 last 19.97) - open/close 1687 (time 1788362156.18 total 63.06 last 22.03) - open/close 1895 (time 1788362166.99 total 73.88 last 19.23) - open/close 2149 (time 1788362178.00 total 84.88 last 23.08) - open/close 2406 (time 1788362188.05 total 94.93 last 25.57) - open/close 2699 (time 1788362198.68 total 105.56 last 27.57) - open/close 2878 (time 1788362209.04 total 115.93 last 17.27) - open/close 3101 (time 1788362219.73 total 126.61 last 20.87) - open/close 3395 (time 1788362229.97 total 136.86 last 28.70) - open/close 3668 (time 1788362240.21 total 147.09 last 26.67) - open/close 3937 (time 1788362250.51 total 157.39 last 26.11) - open/close 4247 (time 1788362260.82 total 167.70 last 30.07) - open/close 4531 (time 1788362271.19 total 178.07 last 27.40) - open/close 4853 (time 1788362281.82 total 188.70 last 30.28) - open/close 5177 (time 1788362292.31 total 199.19 last 30.89) - open/close 5446 (time 1788362302.31 total 209.19 last 26.89) - open/close 5703 (time 1788362312.31 total 219.20 last 25.70) - open/close 5974 (time 1788362322.77 total 229.66 last 25.91) - open/close 6305 (time 1788362333.59 total 240.48 last 30.59) - open/close 6567 (time 1788362344.03 total 250.91 last 25.10) - open/close 6778 (time 1788362355.92 total 262.81 last 17.74) - open/close 7137 (time 1788362366.94 total 273.82 last 32.59) - open/close 7445 (time 1788362377.37 total 284.25 last 29.52) - open/close 7723 (time 1788362389.15 total 296.03 last 23.61) - open/close 8047 (time 1788362399.15 total 306.03 last 32.39) - open/close 8379 (time 1788362409.75 total 316.63 last 31.32) - open/close 8670 (time 1788362419.98 total 326.87 last 28.43) - open/close 8989 (time 1788362430.87 total 337.75 last 29.31) - open/close 9330 (time 1788362441.68 total 348.57 last 31.53) - open/close 9684 (time 1788362452.63 total 359.51 last 32.35) - open/close 9979 (time 1788362463.06 total 369.95 last 28.27) - open/close 10284 (time 1788362473.75 total 380.63 last 28.54) - open/close 10537 (time 1788362484.05 total 390.93 last 24.56) - open/close 10808 (time 1788362494.80 total 401.68 last 25.21) - open/close 10968 (time 1788362504.98 total 411.86 last 15.72) - open/close 11169 (time 1788362515.48 total 422.36 last 19.15) - open/close 11503 (time 1788362525.49 total 432.37 last 33.37) - open/close 12866 (time 1788362535.49 total 442.37 last 136.24) - open/close 14354 (time 1788362545.49 total 452.38 last 148.72) - open/close 15575 (time 1788362555.50 total 462.39 last 122.01) - open/close 16777 (time 1788362565.50 total 472.39 last 120.20) - open/close 18243 (time 1788362575.51 total 482.39 last 146.51) - open/close 19935 (time 1788362585.51 total 492.39 last 169.17) - open/close 21584 (time 1788362595.51 total 502.40 last 164.87) - open/close 23739 (time 1788362605.52 total 512.40 last 215.34) - open/close 25676 (time 1788362615.52 total 522.41 last 193.62) - open/close 27467 (time 1788362625.53 total 532.41 last 179.02) - open/close 29106 (time 1788362635.53 total 542.41 last 163.86) - open/close 30000 (time 1788362640.73 total 547.61 last 171.99) - open/close 31676 (time 1788362650.73 total 557.61 last 167.56) - open/close 33224 (time 1788362660.73 total 567.62 last 154.78) - open/close 34795 (time 1788362670.74 total 577.62 last 156.99) - open/close 36772 (time 1788362680.74 total 587.62 last 197.67) - open/close 38200 (time 1788362690.75 total 597.63 last 142.67) - open/close 39033 (time 1788362700.75 total 607.63 last 83.30) - open/close 39902 (time 1788362710.75 total 617.64 last 86.86) - open/close 40557 (time 1788362720.78 total 627.66 last 65.34) - open/close 41243 (time 1788362730.78 total 637.67 last 68.58) - open/close 41879 (time 1788362740.80 total 647.69 last 63.47) - open/close 42536 (time 1788362750.81 total 657.69 last 65.67) - open/close 43307 (time 1788362760.81 total 667.69 last 77.07) - open/close 43982 (time 1788362770.82 total 677.70 last 67.44) - open/close 44749 (time 1788362780.82 total 687.70 last 76.69) - open/close 45582 (time 1788362790.82 total 697.71 last 83.27) - open/close 46496 (time 1788362800.83 total 707.71 last 91.37) - open/close 47462 (time 1788362810.84 total 717.72 last 96.48) - open/close 48357 (time 1788362820.84 total 727.72 last 89.49) - open/close 49188 (time 1788362830.84 total 737.72 last 83.10) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 742.88 seconds: 66.81 ops/second - unlinked 0 (time 1788362839 ; total 0 ; last 0) - unlinked 10000 (time 1788362931 ; total 92 ; last 92) - unlinked 20000 (time 1788362992 ; total 153 ; last 61) - unlinked 30000 (time 1788363048 ; total 209 ; last 56) - unlinked 40000 (time 1788363096 ; total 257 ; last 48) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 309 seconds: 160.618118 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg354-server unloading modules via unload_modules_local on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing unload_modules_local oleg354-server: modules unloaded. pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2