== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:26:43 (1786116403) start mds service on oleg236-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg236-server' oleg236-server: oleg236-server.virtnet: executing load_modules_local oleg236-server: Loading modules from /home/green/git/lustre-release/lustre oleg236-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg236-server: detected 4 online CPUs by sysfs oleg236-server: MODOPTS_LIBCFS= oleg236-server: Force libcfs to create 2 CPU partitions oleg236-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg236-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg236-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg236-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg236-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 604 (time 1786116456.91 total 10.61 last 56.95) - open/close 1163 (time 1786116467.47 total 21.17 last 52.94) - open/close 1817 (time 1786116478.03 total 31.73 last 61.92) - open/close 2412 (time 1786116488.52 total 42.22 last 56.69) - open/close 2915 (time 1786116498.83 total 52.53 last 48.81) - open/close 3608 (time 1786116509.77 total 63.47 last 63.33) - open/close 4283 (time 1786116520.59 total 74.29 last 62.41) - open/close 4831 (time 1786116531.02 total 84.72 last 52.52) - open/close 5388 (time 1786116541.45 total 95.15 last 53.40) - open/close 5934 (time 1786116552.08 total 105.78 last 51.38) - open/close 6558 (time 1786116562.76 total 116.46 last 58.38) - open/close 7024 (time 1786116573.00 total 126.70 last 45.52) - open/close 7711 (time 1786116583.69 total 137.39 last 64.27) - open/close 8335 (time 1786116594.38 total 148.08 last 58.39) - open/close 9029 (time 1786116605.07 total 158.77 last 64.92) - open/close 9557 (time 1786116615.31 total 169.01 last 51.56) - open/close 10000 (time 1786116624.47 total 178.17 last 48.36) - open/close 10746 (time 1786116635.34 total 189.04 last 68.63) - open/close 11473 (time 1786116645.34 total 199.04 last 72.68) - open/close 12230 (time 1786116655.34 total 209.04 last 75.68) - open/close 12747 (time 1786116665.87 total 219.57 last 49.10) - open/close 13395 (time 1786116676.81 total 230.51 last 59.23) - open/close 14157 (time 1786116686.81 total 240.51 last 76.19) - open/close 14779 (time 1786116697.55 total 251.25 last 57.93) - open/close 15410 (time 1786116708.17 total 261.87 last 59.39) - open/close 15894 (time 1786116718.54 total 272.24 last 46.69) - open/close 16395 (time 1786116728.91 total 282.61 last 48.31) - open/close 16892 (time 1786116739.27 total 292.97 last 47.95) - open/close 20000 (time 1786116748.16 total 301.86 last 349.80) - open/close 25312 (time 1786116758.16 total 311.86 last 531.13) - open/close 30000 (time 1786116767.45 total 321.15 last 504.54) - open/close 34606 (time 1786116777.45 total 331.15 last 460.58) - open/close 39296 (time 1786116787.45 total 341.15 last 468.98) - open/close 44495 (time 1786116797.45 total 351.15 last 519.86) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 360.52 seconds: 137.67 ops/second - unlinked 0 (time 1786116808 ; total 0 ; last 0) - unlinked 10000 (time 1786116819 ; total 11 ; last 11) - unlinked 20000 (time 1786116830 ; total 22 ; last 11) - unlinked 30000 (time 1786116840 ; total 32 ; last 10) - unlinked 40000 (time 1786116851 ; total 43 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 53 seconds: 936.433960 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg236-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg236-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg236-server stop mds service on oleg236-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg236-server stop mds service on oleg236-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg236-server unloading modules via unload_modules_local on: 'oleg236-server' oleg236-server: oleg236-server.virtnet: executing unload_modules_local oleg236-server: modules unloaded. pdsh@oleg236-client: oleg236-client: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-server: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-client: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-server: ssh exited with exit code 2