== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:55:53 (1770688553) start mds service on oleg234-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg234-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 425 (time 1770688610.61 total 10.03 last 42.39) - open/close 667 (time 1770688621.42 total 20.84 last 22.38) - open/close 866 (time 1770688631.79 total 31.21 last 19.18) - open/close 1179 (time 1770688642.54 total 41.96 last 29.12) - open/close 1325 (time 1770688652.56 total 51.97 last 14.58) - open/close 1568 (time 1770688663.22 total 62.63 last 22.79) - open/close 1756 (time 1770688673.58 total 73.00 last 18.14) - open/close 2085 (time 1770688683.59 total 83.00 last 32.89) - open/close 2227 (time 1770688693.62 total 93.03 last 14.15) - open/close 2370 (time 1770688704.56 total 103.98 last 13.07) - open/close 2465 (time 1770688715.31 total 114.73 last 8.84) - open/close 2724 (time 1770688725.69 total 125.10 last 24.96) - open/close 2972 (time 1770688736.56 total 135.97 last 22.82) - open/close 3260 (time 1770688746.74 total 146.15 last 28.29) - open/close 3544 (time 1770688756.74 total 156.15 last 28.40) - open/close 3920 (time 1770688767.42 total 166.84 last 35.20) - open/close 4275 (time 1770688777.42 total 176.84 last 35.48) - open/close 4586 (time 1770688787.95 total 187.37 last 29.54) - open/close 4889 (time 1770688798.83 total 198.25 last 27.86) - open/close 5170 (time 1770688809.78 total 209.19 last 25.67) - open/close 5386 (time 1770688820.27 total 219.69 last 20.58) - open/close 5490 (time 1770688831.09 total 230.51 last 9.61) - open/close 5694 (time 1770688843.57 total 242.98 last 16.35) - open/close 5857 (time 1770688853.68 total 253.10 last 16.11) - open/close 5964 (time 1770688863.69 total 263.10 last 10.70) - open/close 6219 (time 1770688873.71 total 273.13 last 25.43) - open/close 6465 (time 1770688884.47 total 283.89 last 22.86) - open/close 6817 (time 1770688895.41 total 294.82 last 32.19) - open/close 7292 (time 1770688905.41 total 304.83 last 47.49) - open/close 7579 (time 1770688916.08 total 315.50 last 26.90) - open/close 7852 (time 1770688926.45 total 325.86 last 26.33) - open/close 8135 (time 1770688936.94 total 336.36 last 26.96) - open/close 8316 (time 1770688946.99 total 346.41 last 18.01) - open/close 8500 (time 1770688957.23 total 356.65 last 17.97) - open/close 8643 (time 1770688967.28 total 366.70 last 14.22) - open/close 8894 (time 1770688977.29 total 376.71 last 25.08) - open/close 10000 (time 1770688984.25 total 383.66 last 159.04) - open/close 12055 (time 1770688994.25 total 393.67 last 205.44) - open/close 14804 (time 1770689004.25 total 403.67 last 274.90) - open/close 17691 (time 1770689014.25 total 413.67 last 288.63) - open/close 20000 (time 1770689023.12 total 422.54 last 260.30) - open/close 22397 (time 1770689033.12 total 432.54 last 239.69) - open/close 25109 (time 1770689043.13 total 442.55 last 271.02) - open/close 27374 (time 1770689053.13 total 452.55 last 226.49) - open/close 29590 (time 1770689063.13 total 462.55 last 221.58) - open/close 32038 (time 1770689073.14 total 472.56 last 244.60) - open/close 34839 (time 1770689083.14 total 482.56 last 280.06) - open/close 37629 (time 1770689093.15 total 492.57 last 278.72) - open/close 40000 (time 1770689101.57 total 500.99 last 281.43) - open/close 42744 (time 1770689111.58 total 511.00 last 274.23) - open/close 46019 (time 1770689121.58 total 521.00 last 327.46) - open/close 49609 (time 1770689131.59 total 531.00 last 358.86) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 531.06 seconds: 93.46 ops/second - unlinked 0 (time 1770689133 ; total 0 ; last 0) - unlinked 10000 (time 1770689146 ; total 13 ; last 13) - unlinked 20000 (time 1770689161 ; total 28 ; last 15) - unlinked 30000 (time 1770689179 ; total 46 ; last 18) - unlinked 40000 (time 1770689196 ; total 63 ; last 17) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 78 seconds: 636.294861 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg234-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2