== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:18:43 (1787102323) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg152-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg152-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 385 (time 1787102386.40 total 10.84 last 35.50) - open/close 762 (time 1787102397.34 total 21.78 last 34.46) - open/close 1121 (time 1787102409.25 total 33.69 last 30.14) - open/close 1317 (time 1787102419.26 total 43.70 last 19.59) - open/close 1675 (time 1787102432.10 total 56.54 last 27.87) - open/close 1906 (time 1787102442.34 total 66.78 last 22.55) - open/close 2393 (time 1787102452.34 total 76.78 last 48.70) - open/close 2836 (time 1787102462.43 total 86.87 last 43.90) - open/close 3012 (time 1787102472.43 total 96.87 last 17.60) - open/close 3512 (time 1787102482.43 total 106.88 last 50.00) - open/close 3703 (time 1787102492.64 total 117.08 last 18.71) - open/close 4212 (time 1787102502.75 total 127.19 last 50.35) - open/close 4504 (time 1787102513.25 total 137.69 last 27.81) - open/close 4857 (time 1787102523.87 total 148.31 last 33.24) - open/close 5337 (time 1787102533.87 total 158.31 last 48.00) - open/close 5838 (time 1787102543.90 total 168.34 last 49.94) - open/close 6253 (time 1787102554.72 total 179.16 last 38.37) - open/close 6690 (time 1787102565.41 total 189.85 last 40.87) - open/close 7039 (time 1787102575.41 total 199.85 last 34.90) - open/close 7238 (time 1787102585.83 total 210.27 last 19.10) - open/close 7493 (time 1787102595.89 total 220.33 last 25.36) - open/close 7803 (time 1787102605.89 total 230.33 last 30.99) - open/close 8019 (time 1787102615.98 total 240.42 last 21.40) - open/close 8290 (time 1787102626.59 total 251.03 last 25.55) - open/close 8562 (time 1787102636.60 total 261.04 last 27.18) - open/close 8793 (time 1787102649.51 total 273.95 last 17.89) - open/close 9014 (time 1787102659.56 total 284.00 last 22.00) - open/close 9257 (time 1787102670.30 total 294.74 last 22.61) - open/close 9384 (time 1787102680.36 total 304.80 last 12.63) - open/close 9495 (time 1787102690.41 total 314.85 last 11.04) - open/close 9865 (time 1787102700.41 total 324.85 last 36.99) - open/close 10000 (time 1787102706.66 total 331.10 last 21.61) - open/close 10234 (time 1787102717.41 total 341.85 last 21.75) - open/close 10490 (time 1787102727.72 total 352.16 last 24.84) - open/close 10812 (time 1787102738.40 total 362.84 last 30.15) - open/close 10983 (time 1787102750.88 total 375.32 last 13.70) - open/close 11241 (time 1787102761.76 total 386.20 last 23.71) - open/close 13030 (time 1787102771.76 total 396.21 last 178.89) - open/close 15855 (time 1787102781.77 total 406.21 last 282.48) - open/close 19107 (time 1787102791.77 total 416.21 last 325.19) - open/close 20000 (time 1787102794.93 total 419.37 last 282.05) - open/close 22965 (time 1787102804.93 total 429.37 last 296.45) - open/close 26321 (time 1787102814.93 total 439.38 last 335.54) - open/close 30000 (time 1787102824.17 total 448.61 last 398.26) - open/close 33432 (time 1787102834.18 total 458.62 last 343.06) - open/close 37727 (time 1787102844.18 total 468.62 last 429.48) - open/close 40000 (time 1787102848.96 total 473.40 last 475.67) - open/close 44633 (time 1787102858.96 total 483.40 last 463.29) - open/close 48449 (time 1787102868.96 total 493.40 last 381.42) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 496.66 seconds: 99.93 ops/second - unlinked 0 (time 1787102873 ; total 0 ; last 0) - unlinked 10000 (time 1787102892 ; total 19 ; last 19) - unlinked 20000 (time 1787102911 ; total 38 ; last 19) - unlinked 30000 (time 1787102930 ; total 57 ; last 19) - unlinked 40000 (time 1787102942 ; total 69 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 82 seconds: 605.256104 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2