== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:38:02 (1784414282) start mds service on oleg112-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg112-server: detected 4 online CPUs by sysfs oleg112-server: MODOPTS_LIBCFS= oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg112-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg112-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 695 (time 1784414319.98 total 10.04 last 69.24) - open/close 1377 (time 1784414330.99 total 21.05 last 61.95) - open/close 1882 (time 1784414341.36 total 31.41 last 48.71) - open/close 2374 (time 1784414351.73 total 41.78 last 47.44) - open/close 2969 (time 1784414362.29 total 52.34 last 56.36) - open/close 3389 (time 1784414372.46 total 62.52 last 41.27) - open/close 4127 (time 1784414382.64 total 72.69 last 72.53) - open/close 4853 (time 1784414392.75 total 82.81 last 71.80) - open/close 5594 (time 1784414402.93 total 92.98 last 72.80) - open/close 6413 (time 1784414413.17 total 103.22 last 79.98) - open/close 7211 (time 1784414423.47 total 113.53 last 77.46) - open/close 8071 (time 1784414433.97 total 124.02 last 81.94) - open/close 8671 (time 1784414444.92 total 134.97 last 54.81) - open/close 9324 (time 1784414455.73 total 145.78 last 60.39) - open/close 9794 (time 1784414466.16 total 156.21 last 45.06) - open/close 10000 (time 1784414469.64 total 159.70 last 59.15) - open/close 10650 (time 1784414480.69 total 170.75 last 58.82) - open/close 11343 (time 1784414490.93 total 180.98 last 67.70) - open/close 11927 (time 1784414501.62 total 191.68 last 54.61) - open/close 12461 (time 1784414512.11 total 202.17 last 50.92) - open/close 13003 (time 1784414522.61 total 212.67 last 51.62) - open/close 13329 (time 1784414532.66 total 222.71 last 32.46) - open/close 13848 (time 1784414543.09 total 233.14 last 49.75) - open/close 14323 (time 1784414553.46 total 243.51 last 45.80) - open/close 14725 (time 1784414563.63 total 253.69 last 39.52) - open/close 15181 (time 1784414573.88 total 263.93 last 44.52) - open/close 15587 (time 1784414584.05 total 274.10 last 39.91) - open/close 15931 (time 1784414594.05 total 284.10 last 34.40) - open/close 16322 (time 1784414604.08 total 294.14 last 38.96) - open/close 19814 (time 1784414614.09 total 304.14 last 349.16) - open/close 24259 (time 1784414624.09 total 314.14 last 444.46) - open/close 28748 (time 1784414634.09 total 324.15 last 448.66) - open/close 30000 (time 1784414636.91 total 326.96 last 444.62) - open/close 34484 (time 1784414646.91 total 336.96 last 448.36) - open/close 38701 (time 1784414656.91 total 346.96 last 421.66) - open/close 40000 (time 1784414660.02 total 350.07 last 417.96) - open/close 44363 (time 1784414670.02 total 360.07 last 436.28) - open/close 48813 (time 1784414680.02 total 370.07 last 444.94) total: 50000 open/close in 372.40 seconds: 134.26 ops/second - unlinked 0 (time 1784414683 ; total 0 ; last 0) - unlinked 10000 (time 1784414697 ; total 14 ; last 14) - unlinked 20000 (time 1784414712 ; total 29 ; last 15) - unlinked 30000 (time 1784414725 ; total 42 ; last 13) - unlinked 40000 (time 1784414739 ; total 56 ; last 14) total: 50000 unlinks in 72 seconds: 694.444458 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local oleg112-server: modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2