== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:28:05 (1777595285) start mds service on oleg328-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg328-server: detected 4 online CPUs by sysfs oleg328-server: MODOPTS_LIBCFS= oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg328-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg328-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg328-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 442 (time 1777595340.66 total 10.47 last 42.21) - open/close 791 (time 1777595351.41 total 21.22 last 32.46) - open/close 1428 (time 1777595362.23 total 32.04 last 58.90) - open/close 1946 (time 1777595372.53 total 42.34 last 50.28) - open/close 2365 (time 1777595382.71 total 52.52 last 41.17) - open/close 2885 (time 1777595393.20 total 63.01 last 49.55) - open/close 3381 (time 1777595403.44 total 73.25 last 48.43) - open/close 4133 (time 1777595413.44 total 83.25 last 75.20) - open/close 4975 (time 1777595423.60 total 93.41 last 82.88) - open/close 5768 (time 1777595433.65 total 103.46 last 78.91) - open/close 6559 (time 1777595443.70 total 113.51 last 78.72) - open/close 7340 (time 1777595453.75 total 123.56 last 77.75) - open/close 8116 (time 1777595463.75 total 133.56 last 77.59) - open/close 8841 (time 1777595473.75 total 143.56 last 72.49) - open/close 9578 (time 1777595484.72 total 154.53 last 67.16) - open/close 10000 (time 1777595489.80 total 159.61 last 83.13) - open/close 10798 (time 1777595499.80 total 169.61 last 79.79) - open/close 11506 (time 1777595510.77 total 180.58 last 64.53) - open/close 12011 (time 1777595521.46 total 191.27 last 47.25) - open/close 12762 (time 1777595531.76 total 201.57 last 72.89) - open/close 13394 (time 1777595541.81 total 211.62 last 62.89) - open/close 14034 (time 1777595552.63 total 222.44 last 59.18) - open/close 14624 (time 1777595563.44 total 233.25 last 54.56) - open/close 15290 (time 1777595573.44 total 243.25 last 66.59) - open/close 15907 (time 1777595584.24 total 254.05 last 57.14) - open/close 16539 (time 1777595594.24 total 264.05 last 63.20) - open/close 16876 (time 1777595604.47 total 274.28 last 32.96) - open/close 17427 (time 1777595615.41 total 285.22 last 50.36) - open/close 18135 (time 1777595625.46 total 295.27 last 70.46) - open/close 18780 (time 1777595635.46 total 305.27 last 64.50) - open/close 19459 (time 1777595646.33 total 316.14 last 62.46) - open/close 20000 (time 1777595655.14 total 324.95 last 61.39) - open/close 22585 (time 1777595665.14 total 334.95 last 258.44) - open/close 26817 (time 1777595675.14 total 344.95 last 423.17) - open/close 30000 (time 1777595682.10 total 351.91 last 457.67) - open/close 34766 (time 1777595692.10 total 361.91 last 476.59) - open/close 39185 (time 1777595702.10 total 371.91 last 441.86) - open/close 43557 (time 1777595712.10 total 381.91 last 437.15) - open/close 47861 (time 1777595722.10 total 391.91 last 430.34) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 395.83 seconds: 125.39 ops/second - unlinked 0 (time 1777595727 ; total 0 ; last 0) - unlinked 10000 (time 1777595741 ; total 14 ; last 14) - unlinked 20000 (time 1777595754 ; total 27 ; last 13) - unlinked 30000 (time 1777595768 ; total 41 ; last 14) - unlinked 40000 (time 1777595780 ; total 53 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 65 seconds: 763.569214 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg328-server unloading modules via unload_modules_local on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local oleg328-server: modules unloaded. pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2