== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:09:01 (1789085341) Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg307-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg307-server oleg307-server: oleg307-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: kmemleak disabled oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: MODOPTS_LIBCFS= oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg307-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg307-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 735 (time 1789085523.53 total 10.00 last 73.48) - open/close 1453 (time 1789085533.54 total 20.01 last 71.77) - open/close 2039 (time 1789085543.54 total 30.01 last 58.57) - open/close 2714 (time 1789085553.55 total 40.01 last 67.48) - open/close 3344 (time 1789085563.56 total 50.03 last 62.92) - open/close 3995 (time 1789085573.56 total 60.03 last 65.06) total: 4500 open/close in 67.91 seconds: 66.26 ops/second - /unlink 1312 (time 1789085595.86 total 10.00 last 131.17) - /unlink 2646 (time 1789085605.87 total 20.01 last 133.33) - /unlink 3973 (time 1789085615.87 total 30.01 last 132.69) total: 4500 /unlink in 34.00 seconds: 132.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 683 (time 1789085634.42 total 10.01 last 68.22) - open/close 1331 (time 1789085644.43 total 20.02 last 64.74) - open/close 2025 (time 1789085654.43 total 30.02 last 69.40) - open/close 2704 (time 1789085664.44 total 40.03 last 67.82) - open/close 3372 (time 1789085674.46 total 50.05 last 66.67) - open/close 4080 (time 1789085684.47 total 60.06 last 70.77) total: 4500 open/close in 65.86 seconds: 68.32 ops/second - /unlink 1320 (time 1789085704.31 total 10.01 last 131.93) - /unlink 2639 (time 1789085714.31 total 20.01 last 131.90) - /unlink 4025 (time 1789085724.32 total 30.01 last 138.52) total: 4500 /unlink in 33.70 seconds: 133.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 726 (time 1789085742.52 total 10.01 last 72.53) - open/close 1476 (time 1789085752.53 total 20.02 last 74.89) - open/close 2159 (time 1789085762.54 total 30.03 last 68.29) - open/close 2824 (time 1789085772.55 total 40.04 last 66.39) - open/close 3423 (time 1789085782.55 total 50.04 last 59.90) - open/close 4179 (time 1789085792.56 total 60.05 last 75.57) total: 4500 open/close in 64.37 seconds: 69.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1337 (time 1789085811.05 total 10.02 last 133.43) - /unlink 2572 (time 1789085821.06 total 20.03 last 123.42) - /unlink 4050 (time 1789085831.06 total 30.03 last 147.78) total: 4500 /unlink in 33.45 seconds: 134.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 703 (time 1789085849.10 total 10.01 last 70.25) - open/close 1465 (time 1789085859.11 total 20.02 last 76.12) - open/close 2109 (time 1789085869.12 total 30.03 last 64.36) - open/close 2849 (time 1789085879.13 total 40.03 last 73.94) - open/close 3491 (time 1789085889.14 total 50.04 last 64.13) - open/close 4202 (time 1789085899.15 total 60.06 last 71.00) total: 4500 open/close in 64.19 seconds: 70.11 ops/second - /unlink 1232 (time 1789085917.57 total 10.00 last 123.19) - /unlink 2478 (time 1789085927.58 total 20.01 last 124.52) - /unlink 3875 (time 1789085937.58 total 30.01 last 139.67) total: 4500 /unlink in 35.01 seconds: 128.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 660 (time 1789085957.39 total 10.00 last 65.98) - open/close 1368 (time 1789085967.40 total 20.01 last 70.76) - open/close 2068 (time 1789085977.41 total 30.02 last 69.90) - open/close 2784 (time 1789085987.41 total 40.02 last 71.60) - open/close 3489 (time 1789085997.42 total 50.03 last 70.44) - open/close 4202 (time 1789086007.44 total 60.04 last 71.21) total: 4500 open/close in 65.02 seconds: 69.21 ops/second - /unlink 1148 (time 1789086027.69 total 10.00 last 114.78) - /unlink 2322 (time 1789086037.70 total 20.01 last 117.28) - /unlink 3585 (time 1789086047.70 total 30.02 last 126.23) total: 4500 /unlink in 37.33 seconds: 120.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 661 (time 1789086070.02 total 10.01 last 66.05) - open/close 1392 (time 1789086080.03 total 20.02 last 73.00) - open/close 2076 (time 1789086090.04 total 30.03 last 68.35) - open/close 2707 (time 1789086100.06 total 40.05 last 63.00) - open/close 3326 (time 1789086110.06 total 50.05 last 61.88) - open/close 3991 (time 1789086120.07 total 60.06 last 66.45) total: 4500 open/close in 66.84 seconds: 67.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1274 (time 1789086141.70 total 10.00 last 127.37) - /unlink 2475 (time 1789086151.70 total 20.00 last 120.09) - /unlink 3804 (time 1789086161.70 total 30.00 last 132.89) total: 4500 /unlink in 35.94 seconds: 125.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 643 (time 1789086182.33 total 10.00 last 64.29) - open/close 1273 (time 1789086192.33 total 20.01 last 62.97) - open/close 1956 (time 1789086202.34 total 30.02 last 68.22) - open/close 2733 (time 1789086212.34 total 40.02 last 77.70) - open/close 3433 (time 1789086222.35 total 50.02 last 69.97) - open/close 4148 (time 1789086232.35 total 60.03 last 71.46) total: 4500 open/close in 64.64 seconds: 69.62 ops/second - /unlink 1407 (time 1789086250.67 total 10.01 last 140.63) - /unlink 2832 (time 1789086260.67 total 20.01 last 142.46) - /unlink 4128 (time 1789086270.67 total 30.01 last 129.56) total: 4500 /unlink in 33.12 seconds: 135.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 665 (time 1789086288.24 total 10.00 last 66.47) - open/close 1358 (time 1789086298.24 total 20.01 last 69.28) - open/close 1999 (time 1789086308.24 total 30.01 last 64.09) - open/close 2660 (time 1789086318.24 total 40.01 last 66.08) - open/close 3367 (time 1789086328.25 total 50.01 last 70.68) - open/close 4146 (time 1789086338.26 total 60.03 last 77.80) total: 4500 open/close in 64.88 seconds: 69.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1343 (time 1789086357.39 total 10.01 last 134.20) - /unlink 2675 (time 1789086367.39 total 20.01 last 133.18) - /unlink 3939 (time 1789086377.39 total 30.01 last 126.39) total: 4500 /unlink in 34.32 seconds: 131.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 680 (time 1789086396.41 total 10.00 last 67.97) - open/close 1352 (time 1789086406.42 total 20.02 last 67.10) - open/close 2017 (time 1789086416.42 total 30.02 last 66.49) - open/close 2683 (time 1789086426.44 total 40.04 last 66.48) - open/close 3341 (time 1789086436.45 total 50.04 last 65.77) - open/close 4044 (time 1789086446.46 total 60.06 last 70.21) total: 4500 open/close in 67.07 seconds: 67.09 ops/second - /unlink 1317 (time 1789086468.58 total 10.00 last 131.65) - /unlink 2629 (time 1789086478.59 total 20.01 last 131.06) - /unlink 3830 (time 1789086488.59 total 30.02 last 120.04) total: 4500 /unlink in 35.31 seconds: 127.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 661 (time 1789086509.20 total 10.01 last 66.01) - open/close 1359 (time 1789086519.20 total 20.02 last 69.76) - open/close 2106 (time 1789086529.22 total 30.04 last 74.56) - open/close 2808 (time 1789086539.23 total 40.05 last 70.15) - open/close 3510 (time 1789086549.26 total 50.07 last 70.02) - open/close 4092 (time 1789086559.26 total 60.07 last 58.19) total: 4500 open/close in 69.08 seconds: 65.15 ops/second - /unlink 1274 (time 1789086584.59 total 10.00 last 127.39) - /unlink 2476 (time 1789086594.59 total 20.00 last 120.19) - /unlink 3774 (time 1789086604.60 total 30.00 last 129.78) total: 4500 /unlink in 35.52 seconds: 126.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 720 (time 1789086623.99 total 10.00 last 71.98) - open/close 1408 (time 1789086633.99 total 20.01 last 68.76) - open/close 2056 (time 1789086644.00 total 30.02 last 64.74) - open/close 2711 (time 1789086654.00 total 40.02 last 65.48) - open/close 3401 (time 1789086664.02 total 50.04 last 68.87) - open/close 4129 (time 1789086674.02 total 60.04 last 72.79) total: 4500 open/close in 65.17 seconds: 69.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1478 (time 1789086694.04 total 10.01 last 147.66) - /unlink 2921 (time 1789086704.05 total 20.02 last 144.18) - /unlink 4363 (time 1789086714.05 total 30.02 last 144.15) total: 4500 /unlink in 31.00 seconds: 145.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 710 (time 1789086728.80 total 10.00 last 70.98) - open/close 1437 (time 1789086738.80 total 20.01 last 72.67) - open/close 2190 (time 1789086748.81 total 30.02 last 75.22) - open/close 2925 (time 1789086758.82 total 40.03 last 73.40) - open/close 3582 (time 1789086768.85 total 50.05 last 65.56) - open/close 4245 (time 1789086778.85 total 60.06 last 66.27) total: 4500 open/close in 63.55 seconds: 70.81 ops/second - /unlink 1389 (time 1789086796.31 total 10.00 last 138.88) - /unlink 2633 (time 1789086806.31 total 20.00 last 124.39) - /unlink 3982 (time 1789086816.31 total 30.01 last 134.86) total: 4500 /unlink in 33.84 seconds: 133.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 720 (time 1789086835.32 total 10.01 last 71.93) - open/close 1456 (time 1789086845.33 total 20.03 last 73.49) - open/close 2212 (time 1789086855.34 total 30.03 last 75.57) - open/close 2921 (time 1789086865.35 total 40.04 last 70.83) - open/close 3613 (time 1789086875.36 total 50.05 last 69.15) - open/close 4241 (time 1789086885.36 total 60.05 last 62.80) total: 4500 open/close in 63.63 seconds: 70.72 ops/second - /unlink 1350 (time 1789086902.94 total 10.01 last 134.93) - /unlink 2589 (time 1789086912.94 total 20.01 last 123.90) - /unlink 3736 (time 1789086922.94 total 30.01 last 114.66) total: 4500 /unlink in 35.57 seconds: 126.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 716 (time 1789086943.29 total 10.01 last 71.50) - open/close 1419 (time 1789086953.29 total 20.02 last 70.29) - open/close 2105 (time 1789086963.29 total 30.02 last 68.59) - open/close 2839 (time 1789086973.29 total 40.02 last 73.39) - open/close 3573 (time 1789086983.30 total 50.02 last 73.38) - open/close 4239 (time 1789086993.31 total 60.03 last 66.54) total: 4500 open/close in 64.47 seconds: 69.80 ops/second - /unlink 1361 (time 1789087012.58 total 10.00 last 136.09) - /unlink 2656 (time 1789087022.59 total 20.01 last 129.35) - /unlink 3951 (time 1789087032.59 total 30.01 last 129.49) total: 4500 /unlink in 34.67 seconds: 129.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 701 (time 1789087052.12 total 10.00 last 70.10) - open/close 1440 (time 1789087062.12 total 20.01 last 73.86) - open/close 2120 (time 1789087072.14 total 30.03 last 67.87) - open/close 2831 (time 1789087082.16 total 40.04 last 70.99) - open/close 3527 (time 1789087092.16 total 50.04 last 69.59) - open/close 4242 (time 1789087102.16 total 60.05 last 71.47) total: 4500 open/close in 63.54 seconds: 70.82 ops/second - /unlink 1382 (time 1789087119.46 total 10.00 last 138.17) - /unlink 2677 (time 1789087129.47 total 20.00 last 129.47) - /unlink 4012 (time 1789087139.50 total 30.03 last 133.09) total: 4500 /unlink in 34.27 seconds: 131.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 610 (time 1789087157.79 total 10.00 last 60.98) - open/close 1195 (time 1789087167.81 total 20.02 last 58.43) - open/close 1895 (time 1789087177.81 total 30.02 last 69.95) - open/close 2639 (time 1789087187.82 total 40.03 last 74.35) - open/close 3355 (time 1789087197.83 total 50.04 last 71.54) - open/close 4050 (time 1789087207.84 total 60.05 last 69.42) total: 4500 open/close in 66.42 seconds: 67.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1228 (time 1789087228.65 total 10.01 last 122.74) - /unlink 2396 (time 1789087238.65 total 20.01 last 116.76) - /unlink 3705 (time 1789087248.65 total 30.01 last 130.87) total: 4500 /unlink in 36.48 seconds: 123.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 655 (time 1789087269.53 total 10.00 last 65.49) - open/close 1313 (time 1789087279.56 total 20.03 last 65.61) - open/close 1947 (time 1789087289.59 total 30.06 last 63.22) - open/close 2614 (time 1789087299.61 total 40.07 last 66.61) - open/close 3300 (time 1789087309.61 total 50.08 last 68.58) - open/close 3999 (time 1789087319.62 total 60.09 last 69.82) total: 4500 open/close in 67.68 seconds: 66.49 ops/second - /unlink 1312 (time 1789087341.51 total 10.02 last 130.99) - /unlink 2637 (time 1789087351.51 total 20.02 last 132.46) - /unlink 3957 (time 1789087361.51 total 30.02 last 131.96) total: 4500 /unlink in 34.11 seconds: 131.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 724 (time 1789087379.77 total 10.00 last 72.36) - open/close 1493 (time 1789087389.77 total 20.01 last 76.86) - open/close 2191 (time 1789087399.78 total 30.02 last 69.76) - open/close 2751 (time 1789087409.79 total 40.03 last 55.95) - open/close 3242 (time 1789087419.80 total 50.03 last 49.08) - open/close 3862 (time 1789087429.80 total 60.04 last 61.97) total: 4500 open/close in 69.13 seconds: 65.10 ops/second - /unlink 1467 (time 1789087453.46 total 10.00 last 146.65) - /unlink 2741 (time 1789087463.47 total 20.01 last 127.34) - /unlink 4025 (time 1789087473.48 total 30.02 last 128.29) total: 4500 /unlink in 33.61 seconds: 133.90 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95160 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server unloading modules via unload_modules_local on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: kmemleak disabled modules unloaded.