== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:36:49 (1767386209) start mds service on oleg440-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 514 (time 1767386251.79 total 10.04 last 51.18) - open/close 1013 (time 1767386261.96 total 20.22 last 49.03) - open/close 1436 (time 1767386272.01 total 30.27 last 42.11) - open/close 1998 (time 1767386282.70 total 40.95 last 52.58) - open/close 2525 (time 1767386293.32 total 51.58 last 49.60) - open/close 3120 (time 1767386304.01 total 62.27 last 55.67) - open/close 3564 (time 1767386314.01 total 72.27 last 44.39) - open/close 4250 (time 1767386324.94 total 83.20 last 62.79) - open/close 4785 (time 1767386335.24 total 93.50 last 51.91) - open/close 5303 (time 1767386345.74 total 104.00 last 49.35) - open/close 5670 (time 1767386359.31 total 117.57 last 27.05) - open/close 6080 (time 1767386369.31 total 127.57 last 40.99) - open/close 6567 (time 1767386379.85 total 138.11 last 46.20) - open/close 7158 (time 1767386390.28 total 148.54 last 56.66) - open/close 7747 (time 1767386401.04 total 159.29 last 54.77) - open/close 8281 (time 1767386411.66 total 169.91 last 50.28) - open/close 8825 (time 1767386421.66 total 179.92 last 54.39) - open/close 9383 (time 1767386432.27 total 190.52 last 52.61) - open/close 9847 (time 1767386442.57 total 200.82 last 45.04) - open/close 10000 (time 1767386445.09 total 203.35 last 60.64) - open/close 10511 (time 1767386455.50 total 213.75 last 49.10) - open/close 11084 (time 1767386466.12 total 224.38 last 53.94) - open/close 11656 (time 1767386476.56 total 234.81 last 54.80) - open/close 12084 (time 1767386486.73 total 244.99 last 42.08) - open/close 12559 (time 1767386497.03 total 255.29 last 46.10) - open/close 13051 (time 1767386507.53 total 265.79 last 46.85) - open/close 13670 (time 1767386518.41 total 276.67 last 56.91) - open/close 14246 (time 1767386529.36 total 287.62 last 52.61) - open/close 14684 (time 1767386539.59 total 297.85 last 42.79) - open/close 15081 (time 1767386549.60 total 307.86 last 39.69) - open/close 15465 (time 1767386559.60 total 317.86 last 38.39) - open/close 15965 (time 1767386569.74 total 327.99 last 49.33) - open/close 16406 (time 1767386579.79 total 338.04 last 43.89) - open/close 16901 (time 1767386590.22 total 348.48 last 47.45) - open/close 17345 (time 1767386600.39 total 358.65 last 43.65) - open/close 17966 (time 1767386610.89 total 369.15 last 59.12) - open/close 18423 (time 1767386620.94 total 379.19 last 45.50) - open/close 18989 (time 1767386631.50 total 389.76 last 53.57) - open/close 19556 (time 1767386642.06 total 400.31 last 53.72) - open/close 20000 (time 1767386651.19 total 409.45 last 48.61) - open/close 20656 (time 1767386661.97 total 420.22 last 60.88) - open/close 22524 (time 1767386671.97 total 430.22 last 186.79) - open/close 25154 (time 1767386681.97 total 440.22 last 262.99) - open/close 27612 (time 1767386691.97 total 450.23 last 245.73) - open/close 29991 (time 1767386701.97 total 460.23 last 237.86) - open/close 32575 (time 1767386711.98 total 470.23 last 258.30) - open/close 35402 (time 1767386721.98 total 480.23 last 282.67) - open/close 38215 (time 1767386731.98 total 490.24 last 281.25) - open/close 40000 (time 1767386739.46 total 497.72 last 238.62) - open/close 42414 (time 1767386749.46 total 507.72 last 241.32) - open/close 44795 (time 1767386759.47 total 517.72 last 238.03) - open/close 47081 (time 1767386769.47 total 527.72 last 228.56) - open/close 49767 (time 1767386779.47 total 537.72 last 268.59) total: 50000 open/close in 538.51 seconds: 92.85 ops/second - unlinked 0 (time 1767386782 ; total 0 ; last 0) - unlinked 10000 (time 1767386801 ; total 19 ; last 19) - unlinked 20000 (time 1767386825 ; total 43 ; last 24) - unlinked 30000 (time 1767386840 ; total 58 ; last 15) - unlinked 40000 (time 1767386856 ; total 74 ; last 16) total: 50000 unlinks in 98 seconds: 510.204071 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2