== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:22:16 (1782487336) start mds service on oleg134-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg134-server: detected 4 online CPUs by sysfs oleg134-server: MODOPTS_LIBCFS= oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg134-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 712 (time 1782487374.69 total 10.56 last 67.45) - open/close 1314 (time 1782487385.00 total 20.87 last 58.39) - open/close 1662 (time 1782487395.05 total 30.91 last 34.64) - open/close 2202 (time 1782487405.16 total 41.02 last 53.41) - open/close 2598 (time 1782487415.84 total 51.71 last 37.05) - open/close 3073 (time 1782487426.08 total 61.95 last 46.39) - open/close 3717 (time 1782487436.39 total 72.25 last 62.49) - open/close 4233 (time 1782487447.40 total 83.27 last 46.86) - open/close 4729 (time 1782487457.64 total 93.50 last 48.44) - open/close 5270 (time 1782487467.81 total 103.68 last 53.17) - open/close 5673 (time 1782487477.82 total 113.69 last 40.28) - open/close 5823 (time 1782487488.87 total 124.74 last 13.57) - open/close 6411 (time 1782487498.87 total 134.74 last 58.79) - open/close 6993 (time 1782487509.03 total 144.90 last 57.28) - open/close 7560 (time 1782487519.03 total 154.90 last 56.69) - open/close 8088 (time 1782487529.20 total 165.07 last 51.91) - open/close 8640 (time 1782487539.30 total 175.17 last 54.66) - open/close 9123 (time 1782487549.42 total 185.28 last 47.77) - open/close 9694 (time 1782487560.36 total 196.22 last 52.18) - open/close 10000 (time 1782487564.39 total 200.25 last 75.99) - open/close 10586 (time 1782487575.40 total 211.26 last 53.22) - open/close 11195 (time 1782487585.52 total 221.39 last 60.15) - open/close 11767 (time 1782487595.62 total 231.49 last 56.63) - open/close 12359 (time 1782487606.63 total 242.49 last 53.79) - open/close 13008 (time 1782487617.57 total 253.44 last 59.29) - open/close 13633 (time 1782487628.51 total 264.38 last 57.13) - open/close 14270 (time 1782487639.39 total 275.26 last 58.55) - open/close 14895 (time 1782487649.39 total 285.26 last 62.50) - open/close 15507 (time 1782487660.39 total 296.26 last 55.66) - open/close 16131 (time 1782487670.44 total 306.30 last 62.11) - open/close 16640 (time 1782487681.19 total 317.06 last 47.34) - open/close 19014 (time 1782487691.19 total 327.06 last 237.39) - open/close 20000 (time 1782487693.67 total 329.54 last 397.63) - open/close 23632 (time 1782487703.67 total 339.54 last 363.15) - open/close 27192 (time 1782487713.67 total 349.54 last 355.99) - open/close 30000 (time 1782487720.32 total 356.19 last 422.17) - open/close 33661 (time 1782487730.32 total 366.19 last 366.09) - open/close 37198 (time 1782487740.32 total 376.19 last 353.65) - open/close 39708 (time 1782487750.33 total 386.19 last 250.94) - open/close 40000 (time 1782487753.10 total 388.96 last 105.44) - open/close 40814 (time 1782487763.11 total 398.98 last 81.29) - open/close 41333 (time 1782487773.14 total 409.01 last 51.75) - open/close 41573 (time 1782487783.18 total 419.04 last 23.91) - open/close 41749 (time 1782487793.61 total 429.48 last 16.87) - open/close 41972 (time 1782487803.64 total 439.51 last 22.24) - open/close 42234 (time 1782487813.65 total 449.52 last 26.17) - open/close 42589 (time 1782487823.69 total 459.55 last 35.38) - open/close 42938 (time 1782487833.69 total 469.55 last 34.90) - open/close 43344 (time 1782487843.69 total 479.56 last 40.59) - open/close 43921 (time 1782487853.69 total 489.56 last 57.68) - open/close 44364 (time 1782487863.71 total 499.57 last 44.24) - open/close 44882 (time 1782487873.71 total 509.58 last 51.79) - open/close 45225 (time 1782487883.73 total 519.60 last 34.22) - open/close 45683 (time 1782487893.75 total 529.61 last 45.74) - open/close 46289 (time 1782487903.76 total 539.62 last 60.55) - open/close 46895 (time 1782487913.78 total 549.65 last 60.45) - open/close 47544 (time 1782487923.79 total 559.65 last 64.86) - open/close 48195 (time 1782487933.79 total 569.66 last 65.09) - open/close 48891 (time 1782487943.79 total 579.66 last 69.59) - open/close 49507 (time 1782487953.80 total 589.67 last 61.52) total: 50000 open/close in 598.11 seconds: 83.60 ops/second - unlinked 0 (time 1782487967 ; total 0 ; last 0) - unlinked 10000 (time 1782488071 ; total 104 ; last 104) - unlinked 20000 (time 1782488166 ; total 199 ; last 95) - unlinked 30000 (time 1782488268 ; total 301 ; last 102) - unlinked 40000 (time 1782488377 ; total 410 ; last 109) total: 50000 unlinks in 511 seconds: 97.847359 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server unloading modules via unload_modules_local on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2