== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:17:12 (1772680632) start mds service on oleg235-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions oleg235-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg235-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg235-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg235-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg235-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 826 (time 1772680690.36 total 10.06 last 82.09) - open/close 1646 (time 1772680700.41 total 20.11 last 81.60) - open/close 2482 (time 1772680710.41 total 30.11 last 83.58) - open/close 3403 (time 1772680720.76 total 40.46 last 88.97) - open/close 4177 (time 1772680730.76 total 50.47 last 77.39) - open/close 4963 (time 1772680740.76 total 60.47 last 78.59) - open/close 5723 (time 1772680750.76 total 70.47 last 75.99) - open/close 6523 (time 1772680760.76 total 80.47 last 79.99) - open/close 7086 (time 1772680771.26 total 90.96 last 53.66) - open/close 7574 (time 1772680781.62 total 101.33 last 47.07) - open/close 8426 (time 1772680791.73 total 111.44 last 84.28) - open/close 9322 (time 1772680802.10 total 121.81 last 86.42) - open/close 10000 (time 1772680809.91 total 129.61 last 86.86) - open/close 10819 (time 1772680819.91 total 139.61 last 81.89) - open/close 11585 (time 1772680830.01 total 149.71 last 75.85) - open/close 12353 (time 1772680840.12 total 159.82 last 75.95) - open/close 13157 (time 1772680850.17 total 169.87 last 80.01) - open/close 13927 (time 1772680860.22 total 179.92 last 76.63) - open/close 14694 (time 1772680870.22 total 189.92 last 76.69) - open/close 15624 (time 1772680880.70 total 200.40 last 88.76) - open/close 16540 (time 1772680891.26 total 210.96 last 86.74) - open/close 17489 (time 1772680901.82 total 221.53 last 89.82) - open/close 18469 (time 1772680912.45 total 232.15 last 92.25) - open/close 19286 (time 1772680922.68 total 242.38 last 79.82) - open/close 20000 (time 1772680930.60 total 250.30 last 90.17) - open/close 20776 (time 1772680940.67 total 260.37 last 77.08) - open/close 21649 (time 1772680951.16 total 270.87 last 83.17) - open/close 22524 (time 1772680961.40 total 281.10 last 85.49) - open/close 23323 (time 1772680971.57 total 291.28 last 78.51) - open/close 24086 (time 1772680981.69 total 301.39 last 75.46) - open/close 24884 (time 1772680991.80 total 311.51 last 78.88) - open/close 27120 (time 1772681001.80 total 321.51 last 223.59) - open/close 30000 (time 1772681008.03 total 327.74 last 462.19) - open/close 34538 (time 1772681018.04 total 337.74 last 453.73) - open/close 39268 (time 1772681028.04 total 347.74 last 472.95) - open/close 43884 (time 1772681038.04 total 357.74 last 461.59) - open/close 48500 (time 1772681048.04 total 367.74 last 461.51) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 370.20 seconds: 134.07 ops/second - unlinked 0 (time 1772681051 ; total 0 ; last 0) - unlinked 10000 (time 1772681064 ; total 13 ; last 13) - unlinked 20000 (time 1772681076 ; total 25 ; last 12) - unlinked 30000 (time 1772681090 ; total 39 ; last 14) - unlinked 40000 (time 1772681102 ; total 51 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 64 seconds: 775.500000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg235-server unloading modules via unload_modules_local on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing unload_modules_local oleg235-server: modules unloaded. pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2