== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:21:20 (1777911680) start mds service on oleg327-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg327-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 261 (time 1777911721.35 total 11.51 last 22.68) - open/close 542 (time 1777911731.46 total 21.62 last 27.78) - open/close 766 (time 1777911741.89 total 32.06 last 21.48) - open/close 905 (time 1777911752.06 total 42.23 last 13.66) - open/close 1077 (time 1777911762.37 total 52.53 last 16.70) - open/close 1227 (time 1777911772.41 total 62.58 last 14.93) - open/close 1472 (time 1777911782.72 total 72.88 last 23.78) - open/close 1835 (time 1777911793.54 total 83.70 last 33.56) - open/close 2059 (time 1777911803.97 total 94.14 last 21.46) - open/close 2164 (time 1777911814.91 total 105.08 last 9.60) - open/close 2497 (time 1777911824.92 total 115.09 last 33.27) - open/close 2849 (time 1777911835.65 total 125.81 last 32.83) - open/close 3234 (time 1777911846.60 total 136.76 last 35.16) - open/close 3521 (time 1777911857.41 total 147.57 last 26.55) - open/close 3867 (time 1777911868.10 total 158.26 last 32.37) - open/close 4243 (time 1777911878.53 total 168.69 last 36.05) - open/close 4644 (time 1777911889.47 total 179.63 last 36.65) - open/close 5126 (time 1777911900.48 total 190.64 last 43.79) - open/close 5723 (time 1777911910.48 total 200.64 last 59.70) - open/close 6168 (time 1777911921.41 total 211.57 last 40.70) - open/close 6587 (time 1777911931.41 total 221.57 last 41.90) - open/close 7174 (time 1777911942.08 total 232.24 last 55.01) - open/close 7716 (time 1777911952.08 total 242.24 last 54.20) - open/close 8124 (time 1777911962.09 total 252.25 last 40.78) - open/close 8460 (time 1777911972.80 total 262.97 last 31.35) - open/close 8927 (time 1777911982.92 total 273.08 last 46.18) - open/close 9378 (time 1777911992.92 total 283.08 last 45.10) - open/close 9848 (time 1777912003.33 total 293.49 last 45.13) - open/close 10000 (time 1777912006.02 total 296.18 last 56.54) - open/close 10273 (time 1777912016.64 total 306.80 last 25.71) - open/close 10822 (time 1777912026.64 total 316.80 last 54.89) - open/close 11386 (time 1777912036.73 total 326.90 last 55.88) - open/close 11990 (time 1777912046.98 total 337.14 last 58.97) - open/close 13165 (time 1777912056.98 total 347.14 last 117.50) - open/close 16895 (time 1777912066.98 total 357.14 last 372.97) - open/close 20000 (time 1777912075.71 total 365.88 last 355.44) - open/close 23150 (time 1777912085.71 total 375.88 last 314.95) - open/close 27155 (time 1777912095.71 total 385.88 last 400.44) - open/close 30000 (time 1777912103.24 total 393.41 last 377.88) - open/close 33745 (time 1777912113.24 total 403.41 last 374.49) - open/close 37494 (time 1777912123.25 total 413.41 last 374.84) - open/close 40000 (time 1777912129.83 total 419.99 last 380.82) - open/close 44606 (time 1777912139.95 total 430.12 last 454.76) - open/close 48761 (time 1777912149.96 total 440.12 last 415.47) total: 50000 open/close in 444.60 seconds: 112.46 ops/second - unlinked 0 (time 1777912155 ; total 0 ; last 0) - unlinked 10000 (time 1777912176 ; total 21 ; last 21) - unlinked 20000 (time 1777912192 ; total 37 ; last 16) - unlinked 30000 (time 1777912208 ; total 53 ; last 16) - unlinked 40000 (time 1777912222 ; total 67 ; last 14) total: 50000 unlinks in 81 seconds: 617.283936 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2