== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:13:07 (1763878387) start mds service on oleg232-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg232-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg232-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 550 (time 1763878443.45 total 10.44 last 52.69) - open/close 1144 (time 1763878454.27 total 21.26 last 54.90) - open/close 1839 (time 1763878465.21 total 32.20 last 63.52) - open/close 2291 (time 1763878475.39 total 42.38 last 44.42) - open/close 2823 (time 1763878485.88 total 52.87 last 50.68) - open/close 3440 (time 1763878496.76 total 63.75 last 56.72) - open/close 4106 (time 1763878507.77 total 74.76 last 60.49) - open/close 4710 (time 1763878518.33 total 85.32 last 57.20) - open/close 5173 (time 1763878528.57 total 95.56 last 45.21) - open/close 5767 (time 1763878539.32 total 106.31 last 55.25) - open/close 6240 (time 1763878549.50 total 116.49 last 46.47) - open/close 6815 (time 1763878560.19 total 127.18 last 53.81) - open/close 7451 (time 1763878570.94 total 137.93 last 59.14) - open/close 8047 (time 1763878581.69 total 148.68 last 55.44) - open/close 8739 (time 1763878592.57 total 159.56 last 63.59) - open/close 9355 (time 1763878603.19 total 170.18 last 58.00) - open/close 9782 (time 1763878614.21 total 181.19 last 38.78) - open/close 10000 (time 1763878618.25 total 185.24 last 53.86) - open/close 10560 (time 1763878628.99 total 195.98 last 52.17) - open/close 11208 (time 1763878639.81 total 206.80 last 59.89) - open/close 11660 (time 1763878649.81 total 216.80 last 45.20) - open/close 12191 (time 1763878660.35 total 227.34 last 50.37) - open/close 12768 (time 1763878670.78 total 237.77 last 55.33) - open/close 13438 (time 1763878681.72 total 248.71 last 61.22) - open/close 14049 (time 1763878692.60 total 259.59 last 56.17) - open/close 14630 (time 1763878703.16 total 270.15 last 55.00) - open/close 15250 (time 1763878713.85 total 280.84 last 58.01) - open/close 15916 (time 1763878724.60 total 291.59 last 61.93) - open/close 16441 (time 1763878735.61 total 302.60 last 47.71) - open/close 16940 (time 1763878745.61 total 312.60 last 49.89) - open/close 17459 (time 1763878755.61 total 322.60 last 51.89) - open/close 17802 (time 1763878765.69 total 332.68 last 34.04) - open/close 18348 (time 1763878776.64 total 343.63 last 49.85) - open/close 18852 (time 1763878787.13 total 354.12 last 48.04) - open/close 19360 (time 1763878797.50 total 364.49 last 49.00) - open/close 19961 (time 1763878807.50 total 374.49 last 60.09) - open/close 20412 (time 1763878817.85 total 384.84 last 43.56) - open/close 22579 (time 1763878827.85 total 394.84 last 216.66) - open/close 27028 (time 1763878837.86 total 404.84 last 444.83) - open/close 30000 (time 1763878844.81 total 411.80 last 427.13) - open/close 34227 (time 1763878854.82 total 421.80 last 422.61) - open/close 38177 (time 1763878864.82 total 431.81 last 394.89) - open/close 40000 (time 1763878869.26 total 436.25 last 410.24) - open/close 44029 (time 1763878879.26 total 446.25 last 402.86) - open/close 47757 (time 1763878889.27 total 456.26 last 372.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 461.31 seconds: 107.59 ops/second - unlinked 0 (time 1763878896 ; total 0 ; last 0) - unlinked 10000 (time 1763878913 ; total 17 ; last 17) - unlinked 20000 (time 1763878932 ; total 36 ; last 19) - unlinked 30000 (time 1763878948 ; total 52 ; last 16) - unlinked 40000 (time 1763878967 ; total 71 ; last 19) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 92 seconds: 539.478271 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg232-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local oleg232-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg232-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2