== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:38:38 (1775050718) start mds service on oleg325-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg325-server: detected 4 online CPUs by sysfs oleg325-server: MODOPTS_LIBCFS= oleg325-server: Force libcfs to create 2 CPU partitions oleg325-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg325-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg325-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg325-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg325-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0000 oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 765 (time 1775050770.80 total 10.00 last 76.50) - open/close 1384 (time 1775050781.69 total 20.89 last 56.86) - open/close 2069 (time 1775050792.44 total 31.64 last 63.70) - open/close 2715 (time 1775050802.61 total 41.81 last 63.50) - open/close 3431 (time 1775050812.67 total 51.87 last 71.22) - open/close 4056 (time 1775050823.48 total 62.68 last 57.80) - open/close 4390 (time 1775050833.48 total 72.68 last 33.40) - open/close 5077 (time 1775050844.47 total 83.67 last 62.51) - open/close 5779 (time 1775050854.47 total 93.67 last 70.19) - open/close 6491 (time 1775050865.33 total 104.53 last 65.55) - open/close 7203 (time 1775050876.28 total 115.48 last 65.06) - open/close 8006 (time 1775050886.28 total 125.48 last 80.29) - open/close 8537 (time 1775050896.82 total 136.02 last 50.36) - open/close 9189 (time 1775050907.83 total 147.03 last 59.23) - open/close 9701 (time 1775050918.26 total 157.46 last 49.09) - open/close 10000 (time 1775050921.97 total 161.17 last 80.69) - open/close 10713 (time 1775050932.79 total 171.99 last 65.89) - open/close 11339 (time 1775050943.42 total 182.62 last 58.91) - open/close 11932 (time 1775050954.04 total 193.24 last 55.83) - open/close 12592 (time 1775050964.73 total 203.93 last 61.73) - open/close 13330 (time 1775050975.54 total 214.74 last 68.24) - open/close 13927 (time 1775050986.10 total 225.30 last 56.54) - open/close 14540 (time 1775050996.66 total 235.86 last 58.04) - open/close 15166 (time 1775051007.29 total 246.49 last 58.94) - open/close 15894 (time 1775051018.23 total 257.43 last 66.49) - open/close 16674 (time 1775051028.24 total 267.44 last 77.99) - open/close 17414 (time 1775051038.24 total 277.44 last 74.00) - open/close 18064 (time 1775051048.96 total 288.16 last 60.63) - open/close 18849 (time 1775051058.96 total 298.16 last 78.50) - open/close 19651 (time 1775051069.94 total 309.15 last 73.00) - open/close 20000 (time 1775051073.86 total 313.06 last 89.04) - open/close 24861 (time 1775051083.87 total 323.07 last 486.06) - open/close 29795 (time 1775051093.87 total 333.07 last 493.35) - open/close 34662 (time 1775051103.87 total 343.07 last 486.65) - open/close 39578 (time 1775051113.87 total 353.07 last 491.56) - open/close 44398 (time 1775051123.87 total 363.07 last 481.94) - open/close 49207 (time 1775051133.87 total 373.07 last 480.80) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 373.97 seconds: 132.72 ops/second - unlinked 0 (time 1775051136 ; total 0 ; last 0) - unlinked 10000 (time 1775051148 ; total 12 ; last 12) - unlinked 20000 (time 1775051160 ; total 24 ; last 12) - unlinked 30000 (time 1775051171 ; total 35 ; last 11) - unlinked 40000 (time 1775051183 ; total 47 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 58 seconds: 855.724121 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg325-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg325-server unloading modules via unload_modules_local on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing unload_modules_local oleg325-server: modules unloaded. pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2