== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:23:05 (1772169785) start mds service on oleg327-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg327-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 747 (time 1772169846.94 total 10.00 last 74.69) - open/close 1050 (time 1772169857.22 total 20.28 last 29.49) - open/close 1521 (time 1772169868.10 total 31.16 last 43.29) - open/close 2017 (time 1772169878.40 total 41.46 last 48.13) - open/close 2583 (time 1772169888.64 total 51.70 last 55.28) - open/close 3060 (time 1772169899.01 total 62.07 last 46.01) - open/close 3522 (time 1772169909.57 total 72.63 last 43.76) - open/close 4064 (time 1772169919.68 total 82.74 last 53.59) - open/close 4540 (time 1772169930.05 total 93.11 last 45.91) - open/close 5030 (time 1772169940.29 total 103.35 last 47.85) - open/close 5398 (time 1772169950.91 total 113.97 last 34.64) - open/close 5848 (time 1772169961.02 total 124.09 last 44.50) - open/close 6394 (time 1772169971.71 total 134.77 last 51.08) - open/close 6935 (time 1772169982.40 total 145.46 last 50.62) - open/close 7572 (time 1772169992.40 total 155.46 last 63.69) - open/close 8148 (time 1772170003.20 total 166.26 last 53.34) - open/close 8777 (time 1772170014.21 total 177.27 last 57.15) - open/close 9295 (time 1772170024.65 total 187.71 last 49.62) - open/close 9778 (time 1772170034.75 total 197.81 last 47.80) - open/close 10000 (time 1772170038.85 total 201.91 last 54.20) - open/close 10422 (time 1772170048.90 total 211.96 last 41.99) - open/close 10880 (time 1772170059.14 total 222.20 last 44.71) - open/close 11394 (time 1772170069.76 total 232.82 last 48.40) - open/close 11841 (time 1772170079.87 total 242.94 last 44.19) - open/close 12246 (time 1772170090.88 total 253.94 last 36.80) - open/close 12868 (time 1772170101.69 total 264.76 last 57.51) - open/close 13309 (time 1772170111.70 total 274.76 last 44.09) - open/close 13779 (time 1772170122.11 total 285.18 last 45.12) - open/close 14402 (time 1772170133.05 total 296.12 last 56.94) - open/close 14980 (time 1772170143.62 total 306.68 last 54.71) - open/close 15545 (time 1772170154.18 total 317.24 last 53.53) - open/close 16061 (time 1772170164.67 total 327.73 last 49.16) - open/close 16495 (time 1772170174.92 total 337.98 last 42.35) - open/close 17044 (time 1772170185.73 total 348.79 last 50.79) - open/close 17543 (time 1772170196.23 total 359.29 last 47.52) - open/close 18083 (time 1772170206.91 total 369.97 last 50.55) - open/close 18558 (time 1772170216.91 total 379.98 last 47.49) - open/close 19004 (time 1772170227.14 total 390.20 last 43.63) - open/close 19521 (time 1772170237.76 total 400.82 last 48.67) - open/close 20000 (time 1772170247.07 total 410.13 last 51.45) - open/close 20687 (time 1772170257.07 total 420.13 last 68.69) - open/close 23145 (time 1772170267.07 total 430.14 last 245.76) - open/close 25409 (time 1772170277.08 total 440.14 last 226.34) - open/close 27629 (time 1772170287.08 total 450.14 last 221.96) - open/close 29864 (time 1772170297.08 total 460.14 last 223.45) - open/close 32103 (time 1772170307.08 total 470.15 last 223.84) - open/close 34526 (time 1772170317.08 total 480.15 last 242.29) - open/close 36728 (time 1772170327.09 total 490.15 last 220.12) - open/close 38932 (time 1772170337.09 total 500.15 last 220.30) - open/close 40000 (time 1772170342.21 total 505.28 last 208.52) - open/close 42170 (time 1772170352.22 total 515.28 last 216.95) - open/close 44348 (time 1772170362.22 total 525.28 last 217.70) - open/close 46441 (time 1772170372.23 total 535.29 last 209.22) - open/close 48635 (time 1772170382.23 total 545.29 last 219.31) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 549.68 seconds: 90.29 ops/second - unlinked 0 (time 1772170388 ; total 0 ; last 0) - unlinked 10000 (time 1772170413 ; total 25 ; last 25) - unlinked 20000 (time 1772170437 ; total 49 ; last 24) - unlinked 30000 (time 1772170461 ; total 73 ; last 24) - unlinked 40000 (time 1772170485 ; total 97 ; last 24) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 119 seconds: 417.075623 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2