== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:51:12 (1781232672) Stopping clients: oleg358-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg358-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg358-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg358-server oleg358-server: oleg358-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg358-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg358-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 681 (time 1781232874.74 total 10.00 last 68.08) - open/close 1353 (time 1781232884.75 total 20.01 last 67.17) - open/close 1974 (time 1781232894.75 total 30.01 last 62.06) - open/close 2648 (time 1781232904.76 total 40.02 last 67.34) - open/close 3310 (time 1781232914.78 total 50.03 last 66.11) - open/close 3859 (time 1781232924.80 total 60.05 last 54.79) - open/close 4383 (time 1781232934.82 total 70.08 last 52.27) total: 4500 open/close in 71.89 seconds: 62.59 ops/second - /unlink 1220 (time 1781232951.21 total 10.01 last 121.91) - /unlink 2425 (time 1781232961.21 total 20.01 last 120.50) - /unlink 3642 (time 1781232971.21 total 30.01 last 121.69) total: 4500 /unlink in 39.07 seconds: 115.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 657 (time 1781232995.65 total 10.00 last 65.69) - open/close 1301 (time 1781233005.66 total 20.00 last 64.40) - open/close 1965 (time 1781233015.66 total 30.01 last 66.38) - open/close 2729 (time 1781233025.66 total 40.01 last 76.39) - open/close 3420 (time 1781233035.69 total 50.04 last 68.89) - open/close 4099 (time 1781233045.69 total 60.04 last 67.89) total: 4500 open/close in 65.47 seconds: 68.73 ops/second - /unlink 1270 (time 1781233066.31 total 10.00 last 126.94) - /unlink 2521 (time 1781233076.33 total 20.02 last 124.86) - /unlink 3562 (time 1781233086.33 total 30.02 last 104.10) total: 4500 /unlink in 38.44 seconds: 117.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 684 (time 1781233110.29 total 10.01 last 68.31) - open/close 1293 (time 1781233120.30 total 20.02 last 60.83) - open/close 1969 (time 1781233130.31 total 30.03 last 67.57) - open/close 2638 (time 1781233140.31 total 40.03 last 66.88) - open/close 3328 (time 1781233150.31 total 50.03 last 68.98) - open/close 4034 (time 1781233160.32 total 60.04 last 70.56) total: 4500 open/close in 66.90 seconds: 67.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1189 (time 1781233182.11 total 10.00 last 118.85) - /unlink 2274 (time 1781233192.11 total 20.00 last 108.50) - /unlink 3517 (time 1781233202.12 total 30.01 last 124.26) total: 4500 /unlink in 37.93 seconds: 118.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 751 (time 1781233225.15 total 10.01 last 74.99) - open/close 1437 (time 1781233235.15 total 20.02 last 68.59) - open/close 2154 (time 1781233245.15 total 30.02 last 71.69) - open/close 2795 (time 1781233255.16 total 40.03 last 64.03) - open/close 3443 (time 1781233265.19 total 50.05 last 64.63) - open/close 4081 (time 1781233275.21 total 60.08 last 63.64) total: 4500 open/close in 66.58 seconds: 67.59 ops/second - /unlink 1299 (time 1781233296.60 total 10.02 last 129.70) - /unlink 2550 (time 1781233306.60 total 20.02 last 125.04) - /unlink 3669 (time 1781233316.61 total 30.03 last 111.82) total: 4500 /unlink in 37.52 seconds: 119.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 640 (time 1781233339.84 total 10.01 last 63.95) - open/close 1284 (time 1781233349.86 total 20.02 last 64.30) - open/close 2020 (time 1781233359.87 total 30.03 last 73.55) - open/close 2731 (time 1781233369.87 total 40.04 last 71.07) - open/close 3387 (time 1781233379.88 total 50.05 last 65.52) - open/close 3966 (time 1781233389.89 total 60.06 last 57.86) - open/close 4470 (time 1781233399.91 total 70.08 last 50.30) total: 4500 open/close in 70.79 seconds: 63.57 ops/second - /unlink 1209 (time 1781233417.01 total 10.00 last 120.88) - /unlink 2469 (time 1781233427.02 total 20.02 last 125.80) - /unlink 3650 (time 1781233437.03 total 30.02 last 118.03) total: 4500 /unlink in 37.02 seconds: 121.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 720 (time 1781233458.58 total 10.01 last 71.93) - open/close 1407 (time 1781233468.60 total 20.02 last 68.60) - open/close 2035 (time 1781233478.61 total 30.04 last 62.72) - open/close 2813 (time 1781233488.61 total 40.04 last 77.80) - open/close 3524 (time 1781233498.62 total 50.05 last 71.03) - open/close 4196 (time 1781233508.62 total 60.05 last 67.18) total: 4500 open/close in 64.19 seconds: 70.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1226 (time 1781233527.58 total 10.01 last 122.52) - /unlink 2466 (time 1781233537.59 total 20.01 last 123.97) - /unlink 3713 (time 1781233547.62 total 30.04 last 124.30) total: 4500 /unlink in 36.89 seconds: 121.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 698 (time 1781233570.47 total 10.00 last 69.78) - open/close 1330 (time 1781233580.48 total 20.02 last 63.11) - open/close 1944 (time 1781233590.51 total 30.04 last 61.25) - open/close 2609 (time 1781233600.52 total 40.05 last 66.44) - open/close 3316 (time 1781233610.52 total 50.06 last 70.66) - open/close 3995 (time 1781233620.53 total 60.07 last 67.82) total: 4500 open/close in 67.00 seconds: 67.17 ops/second - /unlink 1193 (time 1781233642.89 total 10.00 last 119.26) - /unlink 2217 (time 1781233652.89 total 20.01 last 102.37) - /unlink 3490 (time 1781233662.90 total 30.01 last 127.23) total: 4500 /unlink in 38.23 seconds: 117.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 731 (time 1781233686.27 total 10.00 last 73.09) - open/close 1404 (time 1781233696.28 total 20.01 last 67.27) - open/close 2049 (time 1781233706.29 total 30.02 last 64.43) - open/close 2689 (time 1781233716.29 total 40.02 last 63.99) - open/close 3319 (time 1781233726.30 total 50.03 last 62.95) - open/close 3998 (time 1781233736.30 total 60.03 last 67.86) total: 4500 open/close in 66.78 seconds: 67.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1179 (time 1781233758.09 total 10.01 last 117.74) - /unlink 2424 (time 1781233768.09 total 20.02 last 124.45) - /unlink 3606 (time 1781233778.09 total 30.02 last 118.17) total: 4500 /unlink in 37.80 seconds: 119.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 646 (time 1781233801.42 total 10.00 last 64.59) - open/close 1285 (time 1781233811.42 total 20.00 last 63.89) - open/close 1918 (time 1781233821.43 total 30.01 last 63.28) - open/close 2586 (time 1781233831.44 total 40.02 last 66.70) - open/close 3259 (time 1781233841.45 total 50.03 last 67.23) - open/close 3959 (time 1781233851.45 total 60.03 last 69.99) total: 4500 open/close in 68.44 seconds: 65.75 ops/second - /unlink 1249 (time 1781233875.05 total 10.01 last 124.77) - /unlink 2417 (time 1781233885.06 total 20.02 last 116.72) - /unlink 3682 (time 1781233895.06 total 30.02 last 126.43) total: 4500 /unlink in 37.19 seconds: 121.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 617 (time 1781233917.29 total 10.02 last 61.59) - open/close 1264 (time 1781233927.30 total 20.02 last 64.66) - open/close 1928 (time 1781233937.30 total 30.03 last 66.36) - open/close 2586 (time 1781233947.32 total 40.04 last 65.71) - open/close 3271 (time 1781233957.33 total 50.05 last 68.43) - open/close 3983 (time 1781233967.33 total 60.06 last 71.18) total: 4500 open/close in 67.94 seconds: 66.24 ops/second - /unlink 1256 (time 1781233990.18 total 10.01 last 125.49) - /unlink 2525 (time 1781234000.18 total 20.01 last 126.88) - /unlink 3750 (time 1781234010.18 total 30.01 last 122.46) total: 4500 /unlink in 36.63 seconds: 122.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 722 (time 1781234030.97 total 10.01 last 72.10) - open/close 1353 (time 1781234040.97 total 20.02 last 63.07) - open/close 2021 (time 1781234050.98 total 30.03 last 66.75) - open/close 2700 (time 1781234061.00 total 40.04 last 67.80) - open/close 3278 (time 1781234071.00 total 50.04 last 57.78) - open/close 3739 (time 1781234081.04 total 60.09 last 45.89) - open/close 4307 (time 1781234091.05 total 70.09 last 56.76) total: 4500 open/close in 73.06 seconds: 61.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1093 (time 1781234109.02 total 10.01 last 109.14) - /unlink 2300 (time 1781234119.02 total 20.02 last 120.66) - /unlink 3577 (time 1781234129.02 total 30.02 last 127.66) total: 4500 /unlink in 37.73 seconds: 119.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 679 (time 1781234151.07 total 10.02 last 67.77) - open/close 1306 (time 1781234161.08 total 20.03 last 62.65) - open/close 1937 (time 1781234171.09 total 30.04 last 63.03) - open/close 2623 (time 1781234181.09 total 40.04 last 68.55) - open/close 3293 (time 1781234191.10 total 50.05 last 67.00) - open/close 3873 (time 1781234201.11 total 60.06 last 57.91) - open/close 4466 (time 1781234211.13 total 70.08 last 59.20) total: 4500 open/close in 70.57 seconds: 63.76 ops/second - /unlink 1284 (time 1781234226.09 total 10.01 last 128.30) - /unlink 2576 (time 1781234236.10 total 20.02 last 129.09) - /unlink 3759 (time 1781234246.10 total 30.02 last 118.28) total: 4500 /unlink in 37.01 seconds: 121.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 636 (time 1781234268.64 total 10.01 last 63.56) - open/close 1272 (time 1781234278.65 total 20.02 last 63.50) - open/close 1925 (time 1781234288.66 total 30.02 last 65.29) - open/close 2589 (time 1781234298.66 total 40.03 last 66.34) - open/close 3147 (time 1781234308.67 total 50.04 last 55.78) - open/close 3760 (time 1781234318.69 total 60.06 last 61.16) - open/close 4247 (time 1781234328.71 total 70.08 last 48.61) total: 4500 open/close in 73.82 seconds: 60.96 ops/second - /unlink 1225 (time 1781234347.93 total 10.03 last 122.19) - /unlink 2397 (time 1781234357.93 total 20.03 last 117.19) - /unlink 3653 (time 1781234367.95 total 30.04 last 125.41) total: 4500 /unlink in 38.46 seconds: 116.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 648 (time 1781234392.52 total 10.01 last 64.76) - open/close 1308 (time 1781234402.52 total 20.01 last 66.00) - open/close 1929 (time 1781234412.52 total 30.01 last 62.05) - open/close 2598 (time 1781234422.53 total 40.02 last 66.88) - open/close 3273 (time 1781234432.54 total 50.03 last 67.42) - open/close 3879 (time 1781234442.55 total 60.04 last 60.55) - open/close 4436 (time 1781234452.56 total 70.05 last 55.65) total: 4500 open/close in 70.89 seconds: 63.48 ops/second - /unlink 1210 (time 1781234468.58 total 10.00 last 120.95) - /unlink 2448 (time 1781234478.58 total 20.01 last 123.78) - /unlink 3667 (time 1781234488.59 total 30.01 last 121.88) total: 4500 /unlink in 37.14 seconds: 121.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 678 (time 1781234510.56 total 10.00 last 67.79) - open/close 1308 (time 1781234520.57 total 20.01 last 62.94) - open/close 1926 (time 1781234530.57 total 30.01 last 61.79) - open/close 2513 (time 1781234540.58 total 40.02 last 58.68) - open/close 3087 (time 1781234550.58 total 50.02 last 57.36) - open/close 3722 (time 1781234560.59 total 60.03 last 63.44) - open/close 4377 (time 1781234570.60 total 70.04 last 65.46) total: 4500 open/close in 71.99 seconds: 62.51 ops/second - /unlink 1132 (time 1781234587.53 total 10.01 last 113.10) - /unlink 2254 (time 1781234597.54 total 20.01 last 112.16) - /unlink 3338 (time 1781234607.54 total 30.02 last 108.36) - /unlink 4410 (time 1781234617.54 total 40.02 last 107.19) total: 4500 /unlink in 40.78 seconds: 110.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 632 (time 1781234633.51 total 10.01 last 63.15) - open/close 1191 (time 1781234643.51 total 20.01 last 55.88) - open/close 1923 (time 1781234653.51 total 30.01 last 73.20) - open/close 2593 (time 1781234663.52 total 40.02 last 66.97) - open/close 3263 (time 1781234673.52 total 50.02 last 66.99) - open/close 3998 (time 1781234683.53 total 60.03 last 73.45) total: 4500 open/close in 66.88 seconds: 67.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1285 (time 1781234704.87 total 10.01 last 128.43) - /unlink 2613 (time 1781234714.88 total 20.01 last 132.70) - /unlink 3689 (time 1781234724.89 total 30.02 last 107.56) total: 4500 /unlink in 37.22 seconds: 120.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 663 (time 1781234747.15 total 10.00 last 66.28) - open/close 1318 (time 1781234757.16 total 20.01 last 65.48) - open/close 2054 (time 1781234767.16 total 30.01 last 73.58) - open/close 2786 (time 1781234777.17 total 40.02 last 73.14) - open/close 3499 (time 1781234787.18 total 50.03 last 71.20) - open/close 4186 (time 1781234797.18 total 60.03 last 68.70) total: 4500 open/close in 64.35 seconds: 69.93 ops/second - /unlink 1269 (time 1781234816.12 total 10.00 last 126.89) - /unlink 2532 (time 1781234826.12 total 20.00 last 126.29) - /unlink 3563 (time 1781234836.12 total 30.00 last 103.09) - /unlink 4493 (time 1781234846.13 total 40.00 last 92.98) total: 4500 /unlink in 40.04 seconds: 112.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 570 (time 1781234863.46 total 10.01 last 56.95) - open/close 1164 (time 1781234873.46 total 20.01 last 59.37) - open/close 1874 (time 1781234883.47 total 30.02 last 70.96) - open/close 2567 (time 1781234893.48 total 40.02 last 69.26) - open/close 3266 (time 1781234903.48 total 50.03 last 69.87) - open/close 3940 (time 1781234913.48 total 60.03 last 67.37) total: 4500 open/close in 67.48 seconds: 66.69 ops/second - /unlink 1298 (time 1781234936.03 total 10.00 last 129.75) - /unlink 2588 (time 1781234946.03 total 20.01 last 128.95) - /unlink 3867 (time 1781234956.03 total 30.01 last 127.87) total: 4500 /unlink in 35.19 seconds: 127.88 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 94115 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: modules unloaded.