== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:15:21 (1771402521) start mds service on oleg320-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg320-server: detected 4 online CPUs by sysfs oleg320-server: MODOPTS_LIBCFS= oleg320-server: Force libcfs to create 2 CPU partitions oleg320-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg320-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg320-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg320-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 289 (time 1771402592.36 total 11.03 last 26.20) - open/close 451 (time 1771402602.49 total 21.16 last 15.99) - open/close 614 (time 1771402612.50 total 31.17 last 16.29) - open/close 933 (time 1771402622.71 total 41.38 last 31.23) - open/close 1266 (time 1771402633.40 total 52.06 last 31.17) - open/close 1449 (time 1771402643.89 total 62.56 last 17.44) - open/close 1682 (time 1771402654.71 total 73.38 last 21.52) - open/close 1801 (time 1771402666.48 total 85.15 last 10.11) - open/close 1927 (time 1771402677.49 total 96.16 last 11.45) - open/close 2051 (time 1771402688.11 total 106.78 last 11.68) - open/close 2181 (time 1771402698.12 total 116.79 last 12.98) - open/close 2323 (time 1771402709.11 total 127.78 last 12.93) - open/close 2515 (time 1771402720.49 total 139.16 last 16.87) - open/close 2781 (time 1771402730.49 total 149.16 last 26.59) - open/close 2955 (time 1771402741.48 total 160.15 last 15.83) - open/close 3140 (time 1771402751.72 total 170.39 last 18.07) - open/close 3332 (time 1771402763.30 total 181.97 last 16.58) - open/close 3483 (time 1771402773.68 total 192.35 last 14.56) - open/close 3648 (time 1771402784.17 total 202.84 last 15.72) - open/close 3793 (time 1771402794.18 total 212.85 last 14.48) - open/close 3956 (time 1771402805.17 total 223.83 last 14.84) - open/close 4206 (time 1771402815.17 total 233.84 last 24.99) - open/close 4563 (time 1771402825.51 total 244.18 last 34.51) - open/close 4873 (time 1771402835.62 total 254.29 last 30.66) - open/close 5123 (time 1771402846.06 total 264.72 last 23.97) - open/close 5402 (time 1771402856.81 total 275.48 last 25.94) - open/close 5692 (time 1771402867.63 total 286.30 last 26.81) - open/close 5911 (time 1771402878.06 total 296.73 last 21.00) - open/close 6225 (time 1771402889.07 total 307.74 last 28.52) - open/close 6496 (time 1771402899.75 total 318.42 last 25.37) - open/close 6701 (time 1771402910.69 total 329.36 last 18.74) - open/close 6920 (time 1771402921.71 total 340.38 last 19.88) - open/close 7103 (time 1771402932.72 total 351.39 last 16.62) - open/close 7318 (time 1771402943.15 total 361.82 last 20.61) - open/close 7543 (time 1771402953.46 total 372.12 last 21.83) - open/close 7812 (time 1771402963.46 total 382.13 last 26.88) - open/close 7995 (time 1771402973.48 total 392.15 last 18.27) - open/close 8326 (time 1771402984.42 total 403.09 last 30.25) - open/close 8620 (time 1771402995.30 total 413.97 last 27.02) - open/close 8820 (time 1771403005.31 total 423.98 last 19.99) - open/close 9098 (time 1771403015.54 total 434.21 last 27.16) - open/close 9358 (time 1771403026.60 total 445.27 last 23.51) - open/close 9575 (time 1771403037.35 total 456.02 last 20.18) - open/close 9860 (time 1771403047.36 total 466.02 last 28.49) - open/close 10000 (time 1771403053.45 total 472.12 last 22.98) - open/close 10211 (time 1771403064.04 total 482.71 last 19.92) - open/close 10434 (time 1771403074.05 total 492.71 last 22.28) - open/close 12170 (time 1771403084.05 total 502.71 last 173.60) - open/close 14312 (time 1771403094.05 total 512.72 last 214.10) - open/close 16150 (time 1771403104.05 total 522.72 last 183.75) - open/close 18303 (time 1771403114.06 total 532.72 last 215.27) - open/close 20000 (time 1771403121.60 total 540.27 last 224.87) - open/close 21653 (time 1771403131.60 total 550.27 last 165.29) - open/close 23134 (time 1771403141.61 total 560.28 last 148.02) - open/close 24882 (time 1771403151.61 total 570.28 last 174.70) - open/close 26697 (time 1771403161.61 total 580.28 last 181.49) - open/close 28300 (time 1771403171.61 total 590.28 last 160.29) - open/close 30000 (time 1771403180.31 total 598.98 last 195.54) - open/close 31958 (time 1771403190.31 total 608.98 last 195.79) - open/close 33622 (time 1771403200.31 total 618.98 last 166.35) - open/close 35189 (time 1771403210.32 total 628.98 last 156.64) - open/close 37149 (time 1771403220.32 total 638.99 last 195.94) - open/close 38866 (time 1771403230.32 total 648.99 last 171.69) - open/close 40000 (time 1771403238.63 total 657.30 last 136.47) - open/close 41882 (time 1771403248.63 total 667.30 last 188.14) - open/close 43706 (time 1771403258.63 total 677.30 last 182.39) - open/close 45305 (time 1771403268.63 total 687.30 last 159.87) - open/close 47010 (time 1771403278.64 total 697.30 last 170.49) - open/close 48616 (time 1771403288.64 total 707.31 last 160.55) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 713.26 seconds: 69.59 ops/second - unlinked 0 (time 1771403296 ; total 0 ; last 0) - unlinked 10000 (time 1771403331 ; total 35 ; last 35) - unlinked 20000 (time 1771403367 ; total 71 ; last 36) - unlinked 30000 (time 1771403403 ; total 107 ; last 36) - unlinked 40000 (time 1771403430 ; total 134 ; last 27) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 165 seconds: 300.799988 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server unloading modules via unload_modules_local on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local oleg320-server: modules unloaded. pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2