== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:15:45 (1788930945) Stopping clients: oleg249-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg249-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg249-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg249-server oleg249-server: oleg249-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg249-server: detected 4 online CPUs by sysfs oleg249-server: MODOPTS_LIBCFS= oleg249-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg249-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg249-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg249-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 664 (time 1788931191.84 total 10.01 last 66.37) - open/close 1394 (time 1788931201.85 total 20.01 last 72.94) - open/close 2133 (time 1788931211.85 total 30.01 last 73.89) - open/close 2607 (time 1788931221.85 total 40.02 last 47.38) - open/close 3163 (time 1788931231.86 total 50.03 last 55.56) - open/close 3689 (time 1788931241.88 total 60.05 last 52.50) - open/close 4387 (time 1788931251.89 total 70.05 last 69.74) total: 4500 open/close in 71.60 seconds: 62.85 ops/second - /unlink 1344 (time 1788931268.33 total 10.01 last 134.33) - /unlink 2451 (time 1788931278.33 total 20.01 last 110.67) - /unlink 3639 (time 1788931288.34 total 30.02 last 118.71) total: 4500 /unlink in 37.18 seconds: 121.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 574 (time 1788931311.90 total 10.01 last 57.36) - open/close 1224 (time 1788931321.90 total 20.01 last 64.99) - open/close 1820 (time 1788931331.91 total 30.01 last 59.59) - open/close 2395 (time 1788931341.91 total 40.02 last 57.46) - open/close 2937 (time 1788931351.92 total 50.02 last 54.18) - open/close 3724 (time 1788931361.93 total 60.03 last 78.65) - open/close 4322 (time 1788931371.93 total 70.03 last 59.79) total: 4500 open/close in 72.55 seconds: 62.03 ops/second - /unlink 1069 (time 1788931391.45 total 10.00 last 106.89) - /unlink 2242 (time 1788931401.45 total 20.01 last 117.24) - /unlink 3562 (time 1788931411.47 total 30.02 last 131.78) total: 4500 /unlink in 38.18 seconds: 117.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 623 (time 1788931435.83 total 10.03 last 62.12) - open/close 1257 (time 1788931445.83 total 20.03 last 63.40) - open/close 2020 (time 1788931455.85 total 30.05 last 76.13) - open/close 2744 (time 1788931465.86 total 40.05 last 72.38) - open/close 3466 (time 1788931475.86 total 50.06 last 72.17) - open/close 4320 (time 1788931485.87 total 60.07 last 85.31) total: 4500 open/close in 62.31 seconds: 72.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1210 (time 1788931503.03 total 10.00 last 120.95) - /unlink 2372 (time 1788931513.03 total 20.00 last 116.19) - /unlink 3591 (time 1788931523.04 total 30.01 last 121.81) total: 4500 /unlink in 37.53 seconds: 119.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 645 (time 1788931546.94 total 10.00 last 64.49) - open/close 1308 (time 1788931556.95 total 20.01 last 66.26) - open/close 2007 (time 1788931566.95 total 30.01 last 69.88) - open/close 2709 (time 1788931576.95 total 40.01 last 70.18) - open/close 3546 (time 1788931586.96 total 50.02 last 83.65) - open/close 4244 (time 1788931596.97 total 60.03 last 69.73) total: 4500 open/close in 63.55 seconds: 70.80 ops/second - /unlink 1257 (time 1788931615.07 total 10.00 last 125.69) - /unlink 2481 (time 1788931625.07 total 20.00 last 122.40) - /unlink 3859 (time 1788931635.07 total 30.00 last 137.75) total: 4500 /unlink in 36.06 seconds: 124.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 631 (time 1788931657.19 total 10.02 last 62.99) - open/close 1271 (time 1788931667.19 total 20.02 last 64.00) - open/close 1922 (time 1788931677.20 total 30.02 last 65.08) - open/close 2660 (time 1788931687.21 total 40.03 last 73.72) - open/close 3531 (time 1788931697.21 total 50.03 last 87.08) - open/close 4399 (time 1788931707.21 total 60.04 last 86.77) total: 4500 open/close in 61.24 seconds: 73.48 ops/second - /unlink 1610 (time 1788931724.22 total 10.01 last 160.92) - /unlink 3088 (time 1788931734.22 total 20.01 last 147.76) - /unlink 4386 (time 1788931744.23 total 30.01 last 129.74) total: 4500 /unlink in 31.41 seconds: 143.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 729 (time 1788931761.20 total 10.00 last 72.87) - open/close 1347 (time 1788931771.22 total 20.03 last 61.65) - open/close 1931 (time 1788931781.24 total 30.04 last 58.32) - open/close 2494 (time 1788931791.24 total 40.04 last 56.29) - open/close 3392 (time 1788931801.24 total 50.05 last 89.76) - open/close 4151 (time 1788931811.27 total 60.07 last 75.73) total: 4500 open/close in 64.66 seconds: 69.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1302 (time 1788931831.29 total 10.00 last 130.19) - /unlink 2416 (time 1788931841.30 total 20.00 last 111.36) - /unlink 3645 (time 1788931851.30 total 30.00 last 122.89) total: 4500 /unlink in 36.20 seconds: 124.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 660 (time 1788931872.27 total 10.01 last 65.95) - open/close 1407 (time 1788931882.28 total 20.02 last 74.64) - open/close 2223 (time 1788931892.29 total 30.02 last 81.56) - open/close 3069 (time 1788931902.30 total 40.03 last 84.52) - open/close 3787 (time 1788931912.31 total 50.04 last 71.70) - open/close 4421 (time 1788931922.32 total 60.05 last 63.36) total: 4500 open/close in 61.24 seconds: 73.48 ops/second - /unlink 1272 (time 1788931939.19 total 10.02 last 126.92) - /unlink 2692 (time 1788931949.19 total 20.03 last 141.90) - /unlink 4167 (time 1788931959.19 total 30.03 last 147.47) total: 4500 /unlink in 32.29 seconds: 139.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 698 (time 1788931976.37 total 10.02 last 69.64) - open/close 1406 (time 1788931986.39 total 20.04 last 70.69) - open/close 2098 (time 1788931996.40 total 30.05 last 69.13) - open/close 2760 (time 1788932006.41 total 40.06 last 66.10) - open/close 3393 (time 1788932016.41 total 50.07 last 63.29) - open/close 4027 (time 1788932026.44 total 60.09 last 63.27) total: 4500 open/close in 67.20 seconds: 66.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1309 (time 1788932048.62 total 10.01 last 130.73) - /unlink 2794 (time 1788932058.62 total 20.01 last 148.48) - /unlink 4230 (time 1788932068.62 total 30.02 last 143.56) total: 4500 /unlink in 31.77 seconds: 141.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 677 (time 1788932086.14 total 10.00 last 67.67) - open/close 1324 (time 1788932096.15 total 20.01 last 64.67) - open/close 1974 (time 1788932106.15 total 30.01 last 64.99) - open/close 2577 (time 1788932116.16 total 40.02 last 60.23) - open/close 3208 (time 1788932126.16 total 50.02 last 63.09) - open/close 3870 (time 1788932136.17 total 60.04 last 66.11) total: 4500 open/close in 69.31 seconds: 64.92 ops/second - /unlink 1186 (time 1788932160.05 total 10.00 last 118.60) - /unlink 2498 (time 1788932170.05 total 20.00 last 131.15) - /unlink 3897 (time 1788932180.05 total 30.01 last 139.86) total: 4500 /unlink in 34.92 seconds: 128.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 862 (time 1788932199.19 total 10.00 last 86.19) - open/close 1647 (time 1788932209.20 total 20.01 last 78.44) - open/close 2380 (time 1788932219.21 total 30.02 last 73.24) - open/close 3145 (time 1788932229.21 total 40.02 last 76.49) - open/close 3544 (time 1788932239.22 total 50.03 last 39.87) - open/close 4045 (time 1788932249.24 total 60.05 last 50.00) total: 4500 open/close in 65.97 seconds: 68.21 ops/second - /unlink 1486 (time 1788932270.52 total 10.00 last 148.56) - /unlink 2887 (time 1788932280.52 total 20.00 last 140.10) - /unlink 4195 (time 1788932290.52 total 30.01 last 130.76) total: 4500 /unlink in 32.41 seconds: 138.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 653 (time 1788932308.86 total 10.00 last 65.29) - open/close 1348 (time 1788932318.86 total 20.00 last 69.48) - open/close 2057 (time 1788932328.87 total 30.01 last 70.87) - open/close 2694 (time 1788932338.87 total 40.01 last 63.68) - open/close 3189 (time 1788932348.88 total 50.02 last 49.46) - open/close 3654 (time 1788932358.91 total 60.05 last 46.36) - open/close 4209 (time 1788932368.91 total 70.05 last 55.48) total: 4500 open/close in 73.50 seconds: 61.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1249 (time 1788932388.01 total 10.01 last 124.82) - /unlink 2548 (time 1788932398.01 total 20.01 last 129.90) - /unlink 3871 (time 1788932408.01 total 30.01 last 132.28) total: 4500 /unlink in 35.18 seconds: 127.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 777 (time 1788932429.34 total 10.00 last 77.68) - open/close 1492 (time 1788932439.35 total 20.01 last 71.47) - open/close 2135 (time 1788932449.35 total 30.01 last 64.30) - open/close 2762 (time 1788932459.36 total 40.02 last 62.63) - open/close 3432 (time 1788932469.36 total 50.02 last 66.99) - open/close 3987 (time 1788932479.37 total 60.03 last 55.43) total: 4500 open/close in 68.08 seconds: 66.10 ops/second - /unlink 1221 (time 1788932503.16 total 10.01 last 122.02) - /unlink 2459 (time 1788932513.17 total 20.01 last 123.73) - /unlink 3791 (time 1788932523.17 total 30.02 last 133.16) total: 4500 /unlink in 35.46 seconds: 126.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 754 (time 1788932544.33 total 10.00 last 75.37) - open/close 1525 (time 1788932554.36 total 20.04 last 76.85) - open/close 2169 (time 1788932564.37 total 30.05 last 64.32) - open/close 2881 (time 1788932574.38 total 40.06 last 71.16) - open/close 3682 (time 1788932584.39 total 50.06 last 80.03) - open/close 4387 (time 1788932594.39 total 60.07 last 70.49) total: 4500 open/close in 61.92 seconds: 72.67 ops/second - /unlink 1639 (time 1788932611.48 total 10.00 last 163.86) - /unlink 2900 (time 1788932621.50 total 20.02 last 125.93) - /unlink 4149 (time 1788932631.51 total 30.02 last 124.79) total: 4500 /unlink in 33.28 seconds: 135.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 719 (time 1788932649.63 total 10.01 last 71.82) - open/close 1521 (time 1788932659.64 total 20.02 last 80.16) - open/close 2258 (time 1788932669.65 total 30.03 last 73.61) - open/close 2936 (time 1788932679.65 total 40.03 last 67.78) - open/close 3652 (time 1788932689.65 total 50.03 last 71.60) - open/close 4365 (time 1788932699.66 total 60.04 last 71.25) total: 4500 open/close in 61.56 seconds: 73.10 ops/second - /unlink 1107 (time 1788932717.20 total 10.02 last 110.43) - /unlink 1906 (time 1788932727.21 total 20.03 last 79.83) - /unlink 2948 (time 1788932737.23 total 30.05 last 103.97) - /unlink 4024 (time 1788932747.23 total 40.06 last 107.55) total: 4500 /unlink in 45.27 seconds: 99.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 426 (time 1788932770.31 total 10.02 last 42.51) - open/close 1038 (time 1788932780.32 total 20.03 last 61.14) - open/close 1680 (time 1788932790.33 total 30.04 last 64.15) - open/close 2308 (time 1788932800.34 total 40.04 last 62.78) - open/close 2903 (time 1788932810.35 total 50.05 last 59.45) - open/close 3484 (time 1788932820.36 total 60.07 last 58.01) - open/close 4126 (time 1788932830.38 total 70.09 last 64.08) total: 4500 open/close in 74.61 seconds: 60.32 ops/second - /unlink 1340 (time 1788932849.64 total 10.01 last 133.88) - /unlink 2582 (time 1788932859.65 total 20.01 last 124.19) - /unlink 3625 (time 1788932869.65 total 30.02 last 104.25) total: 4500 /unlink in 37.31 seconds: 120.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 532 (time 1788932892.57 total 10.01 last 53.15) - open/close 994 (time 1788932902.58 total 20.02 last 46.16) - open/close 1629 (time 1788932912.60 total 30.03 last 63.40) - open/close 2304 (time 1788932922.60 total 40.04 last 67.45) - open/close 2981 (time 1788932932.61 total 50.04 last 67.69) - open/close 3793 (time 1788932942.61 total 60.04 last 81.20) - open/close 4471 (time 1788932952.61 total 70.04 last 67.80) total: 4500 open/close in 70.36 seconds: 63.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1078 (time 1788932967.84 total 10.00 last 107.77) - /unlink 2016 (time 1788932977.84 total 20.01 last 93.76) - /unlink 3064 (time 1788932987.84 total 30.01 last 104.79) - /unlink 4240 (time 1788932997.84 total 40.01 last 117.57) total: 4500 /unlink in 42.12 seconds: 106.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 705 (time 1788933014.80 total 10.01 last 70.43) - open/close 1422 (time 1788933024.81 total 20.02 last 71.63) - open/close 2120 (time 1788933034.81 total 30.02 last 69.78) - open/close 2867 (time 1788933044.83 total 40.04 last 74.60) - open/close 3470 (time 1788933054.84 total 50.05 last 60.25) - open/close 4035 (time 1788933064.84 total 60.05 last 56.50) total: 4500 open/close in 67.90 seconds: 66.27 ops/second - /unlink 1226 (time 1788933087.94 total 10.01 last 122.53) - /unlink 2123 (time 1788933097.96 total 20.02 last 89.58) - /unlink 3253 (time 1788933107.97 total 30.03 last 112.88) total: 4500 /unlink in 39.77 seconds: 113.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 632 (time 1788933133.91 total 10.02 last 63.10) - open/close 1298 (time 1788933143.92 total 20.03 last 66.52) - open/close 2026 (time 1788933153.94 total 30.05 last 72.68) - open/close 2568 (time 1788933163.95 total 40.05 last 54.17) - open/close 3288 (time 1788933173.95 total 50.06 last 71.96) - open/close 3969 (time 1788933183.96 total 60.07 last 68.04) total: 4500 open/close in 67.91 seconds: 66.26 ops/second - /unlink 1240 (time 1788933207.00 total 10.00 last 123.94) - /unlink 2295 (time 1788933217.01 total 20.01 last 105.48) - /unlink 3588 (time 1788933227.01 total 30.01 last 129.26) total: 4500 /unlink in 39.83 seconds: 112.99 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95477 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server unloading modules via unload_modules_local on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local oleg249-server: modules unloaded.