== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:29:32 (1789118972) Stopping clients: oleg336-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg336-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg336-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg336-server oleg336-server: oleg336-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg336-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg336-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 779 (time 1789119164.32 total 10.01 last 77.85) - open/close 1585 (time 1789119174.33 total 20.01 last 80.56) - open/close 2264 (time 1789119184.33 total 30.02 last 67.85) - open/close 2935 (time 1789119194.35 total 40.03 last 67.02) - open/close 3604 (time 1789119204.37 total 50.05 last 66.75) - open/close 4204 (time 1789119214.37 total 60.06 last 59.98) total: 4500 open/close in 64.19 seconds: 70.10 ops/second - /unlink 1425 (time 1789119232.85 total 10.00 last 142.45) - /unlink 2835 (time 1789119242.86 total 20.01 last 140.88) - /unlink 4205 (time 1789119252.86 total 30.02 last 136.88) total: 4500 /unlink in 32.10 seconds: 140.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 528 (time 1789119270.26 total 10.01 last 52.77) - open/close 1292 (time 1789119280.27 total 20.02 last 76.30) - open/close 2126 (time 1789119290.28 total 30.02 last 83.37) - open/close 2710 (time 1789119300.28 total 40.02 last 58.39) - open/close 3412 (time 1789119310.30 total 50.05 last 70.02) - open/close 4035 (time 1789119320.33 total 60.07 last 62.14) total: 4500 open/close in 68.16 seconds: 66.02 ops/second - /unlink 1374 (time 1789119343.59 total 10.01 last 137.28) - /unlink 2771 (time 1789119353.59 total 20.01 last 139.64) - /unlink 4249 (time 1789119363.60 total 30.02 last 147.69) total: 4500 /unlink in 32.02 seconds: 140.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 764 (time 1789119380.51 total 10.01 last 76.34) - open/close 1497 (time 1789119390.52 total 20.02 last 73.23) - open/close 2224 (time 1789119400.53 total 30.02 last 72.65) - open/close 3032 (time 1789119410.53 total 40.03 last 80.77) - open/close 3896 (time 1789119420.54 total 50.03 last 86.36) total: 4500 open/close in 57.38 seconds: 78.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1686 (time 1789119442.17 total 10.01 last 168.36) - /unlink 3274 (time 1789119452.17 total 20.02 last 158.74) total: 4500 /unlink in 28.46 seconds: 158.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 848 (time 1789119474.60 total 10.00 last 84.80) - open/close 1640 (time 1789119484.60 total 20.00 last 79.20) - open/close 2492 (time 1789119494.61 total 30.01 last 85.11) - open/close 3316 (time 1789119504.61 total 40.01 last 82.38) - open/close 4093 (time 1789119514.63 total 50.03 last 77.59) total: 4500 open/close in 55.02 seconds: 81.80 ops/second - /unlink 1474 (time 1789119534.63 total 10.00 last 147.38) - /unlink 3019 (time 1789119544.63 total 20.00 last 154.50) - /unlink 4304 (time 1789119554.63 total 30.00 last 128.48) total: 4500 /unlink in 31.82 seconds: 141.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 717 (time 1789119571.70 total 10.01 last 71.63) - open/close 1453 (time 1789119581.71 total 20.02 last 73.55) - open/close 2196 (time 1789119591.72 total 30.02 last 74.24) - open/close 2937 (time 1789119601.73 total 40.04 last 74.01) - open/close 3655 (time 1789119611.74 total 50.04 last 71.74) - open/close 4397 (time 1789119621.75 total 60.06 last 74.11) total: 4500 open/close in 61.61 seconds: 73.04 ops/second - /unlink 1402 (time 1789119637.61 total 10.00 last 140.15) - /unlink 2915 (time 1789119647.62 total 20.01 last 151.25) - /unlink 4189 (time 1789119657.62 total 30.01 last 127.37) total: 4500 /unlink in 32.08 seconds: 140.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 798 (time 1789119673.35 total 10.00 last 79.79) - open/close 1570 (time 1789119683.35 total 20.00 last 77.17) - open/close 2348 (time 1789119693.36 total 30.01 last 77.74) - open/close 3159 (time 1789119703.37 total 40.02 last 81.06) - open/close 4048 (time 1789119713.37 total 50.02 last 88.88) total: 4500 open/close in 55.22 seconds: 81.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1399 (time 1789119731.95 total 10.00 last 139.86) - /unlink 2966 (time 1789119741.95 total 20.00 last 156.68) - /unlink 4358 (time 1789119751.97 total 30.02 last 139.01) total: 4500 /unlink in 30.87 seconds: 145.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 843 (time 1789119766.95 total 10.01 last 84.24) - open/close 1669 (time 1789119776.95 total 20.01 last 82.60) - open/close 2435 (time 1789119786.96 total 30.01 last 76.57) - open/close 3282 (time 1789119796.96 total 40.01 last 84.69) - open/close 4123 (time 1789119806.96 total 50.02 last 84.07) total: 4500 open/close in 54.58 seconds: 82.44 ops/second - /unlink 1659 (time 1789119825.63 total 10.00 last 165.89) - /unlink 3097 (time 1789119835.63 total 20.00 last 143.79) - /unlink 4317 (time 1789119845.64 total 30.00 last 121.98) total: 4500 /unlink in 31.31 seconds: 143.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 798 (time 1789119861.25 total 10.01 last 79.74) - open/close 1592 (time 1789119871.25 total 20.01 last 79.38) - open/close 2349 (time 1789119881.26 total 30.02 last 75.64) - open/close 3004 (time 1789119891.27 total 40.02 last 65.46) - open/close 3706 (time 1789119901.27 total 50.03 last 70.15) - open/close 4438 (time 1789119911.28 total 60.03 last 73.18) total: 4500 open/close in 61.04 seconds: 73.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1364 (time 1789119926.67 total 10.00 last 136.38) - /unlink 2707 (time 1789119936.67 total 20.01 last 134.25) - /unlink 4151 (time 1789119946.67 total 30.01 last 144.40) total: 4500 /unlink in 32.36 seconds: 139.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 770 (time 1789119963.91 total 10.01 last 76.94) - open/close 1576 (time 1789119973.91 total 20.01 last 80.56) - open/close 2347 (time 1789119983.92 total 30.02 last 77.05) - open/close 3104 (time 1789119993.93 total 40.03 last 75.63) - open/close 3884 (time 1789120003.94 total 50.04 last 77.93) total: 4500 open/close in 57.92 seconds: 77.69 ops/second - /unlink 1524 (time 1789120026.66 total 10.00 last 152.37) - /unlink 3009 (time 1789120036.66 total 20.00 last 148.47) total: 4500 /unlink in 29.62 seconds: 151.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 844 (time 1789120060.99 total 10.00 last 84.36) - open/close 1674 (time 1789120071.00 total 20.01 last 82.94) - open/close 2490 (time 1789120081.00 total 30.02 last 81.54) - open/close 3325 (time 1789120091.01 total 40.02 last 83.49) - open/close 4128 (time 1789120101.01 total 50.03 last 80.25) total: 4500 open/close in 54.69 seconds: 82.28 ops/second - /unlink 1559 (time 1789120119.89 total 10.01 last 155.73) - /unlink 3028 (time 1789120129.90 total 20.01 last 146.89) total: 4500 /unlink in 29.66 seconds: 151.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 665 (time 1789120154.34 total 10.00 last 66.47) - open/close 1441 (time 1789120164.36 total 20.02 last 77.46) - open/close 2211 (time 1789120174.36 total 30.03 last 76.96) - open/close 2969 (time 1789120184.37 total 40.04 last 75.75) - open/close 3746 (time 1789120194.38 total 50.04 last 77.63) - open/close 4361 (time 1789120204.38 total 60.05 last 61.48) total: 4500 open/close in 62.49 seconds: 72.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1176 (time 1789120222.82 total 10.01 last 117.53) - /unlink 2370 (time 1789120232.82 total 20.01 last 119.40) - /unlink 3947 (time 1789120242.83 total 30.02 last 157.49) total: 4500 /unlink in 33.56 seconds: 134.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 878 (time 1789120261.82 total 10.02 last 87.65) - open/close 1704 (time 1789120271.84 total 20.03 last 82.46) - open/close 2576 (time 1789120281.84 total 30.04 last 87.18) - open/close 3360 (time 1789120291.85 total 40.04 last 78.36) - open/close 4182 (time 1789120301.85 total 50.04 last 82.20) total: 4500 open/close in 54.53 seconds: 82.53 ops/second - /unlink 1349 (time 1789120320.67 total 10.00 last 134.88) - /unlink 2932 (time 1789120330.67 total 20.00 last 158.29) - /unlink 4451 (time 1789120340.67 total 30.01 last 151.82) total: 4500 /unlink in 30.49 seconds: 147.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 853 (time 1789120355.34 total 10.00 last 85.26) - open/close 1805 (time 1789120365.34 total 20.01 last 95.19) - open/close 2627 (time 1789120375.34 total 30.01 last 82.19) - open/close 3465 (time 1789120385.35 total 40.01 last 83.74) - open/close 4260 (time 1789120395.35 total 50.02 last 79.48) total: 4500 open/close in 52.66 seconds: 85.45 ops/second - /unlink 1437 (time 1789120411.90 total 10.01 last 143.56) - /unlink 3003 (time 1789120421.90 total 20.02 last 156.50) total: 4500 /unlink in 29.42 seconds: 152.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 818 (time 1789120445.54 total 10.00 last 81.78) - open/close 1553 (time 1789120455.57 total 20.03 last 73.33) - open/close 2172 (time 1789120465.58 total 30.04 last 61.81) - open/close 2869 (time 1789120475.58 total 40.04 last 69.68) - open/close 3684 (time 1789120485.59 total 50.05 last 81.42) - open/close 4462 (time 1789120495.60 total 60.06 last 77.77) total: 4500 open/close in 60.39 seconds: 74.52 ops/second - /unlink 1490 (time 1789120510.25 total 10.00 last 148.93) - /unlink 2867 (time 1789120520.25 total 20.01 last 137.67) - /unlink 4286 (time 1789120530.26 total 30.01 last 141.87) total: 4500 /unlink in 31.63 seconds: 142.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 699 (time 1789120546.71 total 10.01 last 69.83) - open/close 1477 (time 1789120556.71 total 20.01 last 77.79) - open/close 2324 (time 1789120566.72 total 30.02 last 84.61) - open/close 3117 (time 1789120576.73 total 40.03 last 79.25) - open/close 3927 (time 1789120586.74 total 50.04 last 80.94) total: 4500 open/close in 57.49 seconds: 78.27 ops/second - /unlink 1313 (time 1789120608.55 total 10.00 last 131.27) - /unlink 2702 (time 1789120618.55 total 20.01 last 138.84) - /unlink 4095 (time 1789120628.55 total 30.01 last 139.29) total: 4500 /unlink in 32.88 seconds: 136.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 774 (time 1789120646.53 total 10.00 last 77.40) - open/close 1500 (time 1789120656.56 total 20.04 last 72.34) - open/close 2301 (time 1789120666.59 total 30.06 last 79.92) - open/close 3049 (time 1789120676.59 total 40.06 last 74.78) - open/close 3836 (time 1789120686.60 total 50.07 last 78.65) total: 4500 open/close in 57.61 seconds: 78.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1461 (time 1789120709.13 total 10.00 last 146.06) - /unlink 2710 (time 1789120719.14 total 20.01 last 124.81) - /unlink 4121 (time 1789120729.14 total 30.01 last 141.08) total: 4500 /unlink in 32.83 seconds: 137.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 806 (time 1789120746.99 total 10.00 last 80.57) - open/close 1542 (time 1789120757.00 total 20.01 last 73.53) - open/close 2180 (time 1789120767.01 total 30.03 last 63.72) - open/close 2705 (time 1789120777.02 total 40.03 last 52.48) - open/close 3384 (time 1789120787.04 total 50.05 last 67.75) - open/close 4176 (time 1789120797.05 total 60.07 last 79.09) total: 4500 open/close in 63.95 seconds: 70.37 ops/second - /unlink 1161 (time 1789120815.61 total 10.00 last 116.08) - /unlink 2604 (time 1789120825.61 total 20.01 last 144.21) - /unlink 4080 (time 1789120835.61 total 30.01 last 147.59) total: 4500 /unlink in 33.09 seconds: 135.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 779 (time 1789120854.45 total 10.01 last 77.86) - open/close 1598 (time 1789120864.47 total 20.02 last 81.76) - open/close 2408 (time 1789120874.48 total 30.03 last 80.92) - open/close 3196 (time 1789120884.49 total 40.04 last 78.71) - open/close 4043 (time 1789120894.50 total 50.06 last 84.61) total: 4500 open/close in 55.95 seconds: 80.43 ops/second - /unlink 1561 (time 1789120914.62 total 10.00 last 156.04) - /unlink 3121 (time 1789120924.62 total 20.01 last 155.95) - /unlink 4411 (time 1789120934.63 total 30.01 last 128.92) total: 4500 /unlink in 30.68 seconds: 146.68 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95201 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: modules unloaded.