== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:49:16 (1787132956) Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg136-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server oleg136-server: oleg136-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg136-server: detected 4 online CPUs by sysfs oleg136-server: MODOPTS_LIBCFS= oleg136-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg136-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg136-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg136-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 746 (time 1787133180.43 total 10.01 last 74.49) - open/close 1528 (time 1787133190.44 total 20.03 last 78.12) - open/close 2256 (time 1787133200.45 total 30.04 last 72.72) - open/close 2889 (time 1787133210.48 total 40.06 last 63.12) - open/close 3495 (time 1787133220.48 total 50.07 last 60.58) - open/close 4179 (time 1787133230.49 total 60.07 last 68.38) total: 4500 open/close in 64.62 seconds: 69.64 ops/second - /unlink 1402 (time 1787133250.02 total 10.00 last 140.16) - /unlink 2678 (time 1787133260.02 total 20.01 last 127.55) - /unlink 4063 (time 1787133270.02 total 30.01 last 138.49) total: 4500 /unlink in 33.21 seconds: 135.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 643 (time 1787133289.38 total 10.01 last 64.24) - open/close 1325 (time 1787133299.38 total 20.01 last 68.18) - open/close 1839 (time 1787133309.42 total 30.05 last 51.21) - open/close 2310 (time 1787133319.45 total 40.08 last 46.98) - open/close 2710 (time 1787133329.46 total 50.09 last 39.95) - open/close 3183 (time 1787133339.49 total 60.12 last 47.15) - open/close 3856 (time 1787133349.50 total 70.13 last 67.24) total: 4500 open/close in 79.92 seconds: 56.31 ops/second - /unlink 1355 (time 1787133373.48 total 10.00 last 135.48) - /unlink 2599 (time 1787133383.49 total 20.01 last 124.32) - /unlink 3666 (time 1787133393.50 total 30.01 last 106.63) total: 4500 /unlink in 37.67 seconds: 119.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 577 (time 1787133416.87 total 10.00 last 57.68) - open/close 1256 (time 1787133426.88 total 20.02 last 67.81) - open/close 1988 (time 1787133436.89 total 30.02 last 73.15) - open/close 2632 (time 1787133446.89 total 40.02 last 64.39) - open/close 3241 (time 1787133456.90 total 50.04 last 60.84) - open/close 3902 (time 1787133466.91 total 60.04 last 66.06) total: 4500 open/close in 68.95 seconds: 65.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1367 (time 1787133490.65 total 10.00 last 136.66) - /unlink 2690 (time 1787133500.65 total 20.01 last 132.23) - /unlink 4016 (time 1787133510.65 total 30.01 last 132.59) total: 4500 /unlink in 33.66 seconds: 133.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 654 (time 1787133530.13 total 10.01 last 65.36) - open/close 1377 (time 1787133540.14 total 20.01 last 72.26) - open/close 2140 (time 1787133550.15 total 30.02 last 76.25) - open/close 2859 (time 1787133560.15 total 40.02 last 71.87) - open/close 3490 (time 1787133570.18 total 50.05 last 62.93) - open/close 4159 (time 1787133580.20 total 60.07 last 66.74) total: 4500 open/close in 63.85 seconds: 70.47 ops/second - /unlink 1212 (time 1787133599.43 total 10.01 last 121.11) - /unlink 2459 (time 1787133609.43 total 20.01 last 124.66) - /unlink 3599 (time 1787133619.44 total 30.02 last 113.92) total: 4500 /unlink in 38.59 seconds: 116.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 516 (time 1787133644.88 total 10.01 last 51.56) - open/close 1026 (time 1787133654.89 total 20.02 last 50.95) - open/close 1641 (time 1787133664.90 total 30.02 last 61.46) - open/close 2312 (time 1787133674.91 total 40.03 last 67.05) - open/close 3015 (time 1787133684.91 total 50.04 last 70.27) - open/close 3789 (time 1787133694.92 total 60.04 last 77.35) total: 4500 open/close in 69.66 seconds: 64.60 ops/second - /unlink 955 (time 1787133720.70 total 10.00 last 95.46) - /unlink 2511 (time 1787133730.70 total 20.01 last 155.56) - /unlink 4197 (time 1787133740.70 total 30.01 last 168.57) total: 4500 /unlink in 32.51 seconds: 138.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 702 (time 1787133759.01 total 10.00 last 70.17) - open/close 1418 (time 1787133769.02 total 20.01 last 71.54) - open/close 2222 (time 1787133779.03 total 30.02 last 80.31) - open/close 2940 (time 1787133789.04 total 40.03 last 71.75) - open/close 3730 (time 1787133799.04 total 50.03 last 78.98) - open/close 4434 (time 1787133809.05 total 60.04 last 70.36) total: 4500 open/close in 60.82 seconds: 73.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1317 (time 1787133825.09 total 10.00 last 131.68) - /unlink 2777 (time 1787133835.09 total 20.01 last 145.92) - /unlink 4314 (time 1787133845.09 total 30.01 last 153.70) total: 4500 /unlink in 31.19 seconds: 144.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 914 (time 1787133860.48 total 10.01 last 91.27) - open/close 1764 (time 1787133870.48 total 20.02 last 84.94) - open/close 2704 (time 1787133880.49 total 30.03 last 93.95) - open/close 3582 (time 1787133890.49 total 40.03 last 87.75) - open/close 4474 (time 1787133900.53 total 50.07 last 88.91) total: 4500 open/close in 50.36 seconds: 89.35 ops/second - /unlink 1458 (time 1787133915.77 total 10.00 last 145.77) - /unlink 2969 (time 1787133925.77 total 20.01 last 151.00) - /unlink 4301 (time 1787133935.79 total 30.02 last 133.03) total: 4500 /unlink in 31.31 seconds: 143.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 854 (time 1787133951.44 total 10.00 last 85.36) - open/close 1741 (time 1787133961.46 total 20.02 last 88.58) - open/close 2600 (time 1787133971.46 total 30.02 last 85.86) - open/close 3479 (time 1787133981.47 total 40.03 last 87.85) - open/close 4352 (time 1787133991.47 total 50.03 last 87.26) total: 4500 open/close in 51.87 seconds: 86.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1678 (time 1787134007.48 total 10.01 last 167.60) - /unlink 3417 (time 1787134017.48 total 20.01 last 173.88) total: 4500 /unlink in 28.51 seconds: 157.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 818 (time 1787134041.07 total 10.00 last 81.78) - open/close 1722 (time 1787134051.08 total 20.02 last 90.26) - open/close 2609 (time 1787134061.10 total 30.03 last 88.56) - open/close 3361 (time 1787134071.11 total 40.05 last 75.09) - open/close 4150 (time 1787134081.13 total 50.06 last 78.78) total: 4500 open/close in 53.80 seconds: 83.64 ops/second - /unlink 1525 (time 1787134099.12 total 10.00 last 152.44) - /unlink 3282 (time 1787134109.12 total 20.01 last 175.63) total: 4500 /unlink in 26.84 seconds: 167.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1018 (time 1787134129.98 total 10.01 last 101.73) - open/close 1916 (time 1787134139.98 total 20.01 last 89.77) - open/close 2843 (time 1787134150.01 total 30.03 last 92.51) - open/close 3734 (time 1787134160.02 total 40.04 last 88.99) total: 4500 open/close in 47.74 seconds: 94.25 ops/second - /unlink 1354 (time 1787134181.76 total 10.00 last 135.35) - /unlink 3035 (time 1787134191.77 total 20.01 last 167.93) total: 4500 /unlink in 28.91 seconds: 155.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 888 (time 1787134215.42 total 10.01 last 88.74) - open/close 1787 (time 1787134225.42 total 20.01 last 89.88) - open/close 2495 (time 1787134235.44 total 30.03 last 70.65) - open/close 3411 (time 1787134245.45 total 40.03 last 91.59) - open/close 4325 (time 1787134255.47 total 50.05 last 91.21) total: 4500 open/close in 51.65 seconds: 87.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1447 (time 1787134270.95 total 10.00 last 144.65) - /unlink 3005 (time 1787134280.95 total 20.00 last 155.79) total: 4500 /unlink in 28.78 seconds: 156.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 746 (time 1787134305.13 total 10.00 last 74.57) - open/close 1450 (time 1787134315.13 total 20.01 last 70.38) - open/close 2173 (time 1787134325.14 total 30.02 last 72.23) - open/close 3008 (time 1787134335.15 total 40.02 last 83.45) - open/close 3869 (time 1787134345.15 total 50.02 last 86.08) total: 4500 open/close in 59.70 seconds: 75.37 ops/second - /unlink 1059 (time 1787134371.23 total 10.00 last 105.86) - /unlink 2317 (time 1787134381.24 total 20.01 last 125.72) - /unlink 3532 (time 1787134391.25 total 30.02 last 121.45) total: 4500 /unlink in 35.51 seconds: 126.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 630 (time 1787134412.84 total 10.00 last 62.98) - open/close 1105 (time 1787134422.84 total 20.01 last 47.47) - open/close 1839 (time 1787134432.85 total 30.01 last 73.37) - open/close 2680 (time 1787134442.85 total 40.02 last 84.08) - open/close 3446 (time 1787134452.90 total 50.06 last 76.23) - open/close 4221 (time 1787134462.91 total 60.07 last 77.44) total: 4500 open/close in 63.06 seconds: 71.36 ops/second - /unlink 1569 (time 1787134480.38 total 10.01 last 156.77) - /unlink 2845 (time 1787134490.39 total 20.02 last 127.44) - /unlink 3750 (time 1787134500.39 total 30.02 last 90.47) total: 4500 /unlink in 36.84 seconds: 122.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 844 (time 1787134522.75 total 10.00 last 84.37) - open/close 1743 (time 1787134532.76 total 20.01 last 89.84) - open/close 2568 (time 1787134542.76 total 30.01 last 82.47) - open/close 3267 (time 1787134552.77 total 40.02 last 69.88) - open/close 4119 (time 1787134562.78 total 50.03 last 85.05) total: 4500 open/close in 54.69 seconds: 82.28 ops/second - /unlink 1679 (time 1787134581.45 total 10.00 last 167.83) - /unlink 3359 (time 1787134591.46 total 20.01 last 167.95) total: 4500 /unlink in 29.07 seconds: 154.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 687 (time 1787134615.77 total 10.01 last 68.61) - open/close 1377 (time 1787134625.77 total 20.01 last 68.99) - open/close 2185 (time 1787134635.78 total 30.02 last 80.75) - open/close 3084 (time 1787134645.78 total 40.02 last 89.86) - open/close 3951 (time 1787134655.80 total 50.04 last 86.55) total: 4500 open/close in 58.11 seconds: 77.44 ops/second - /unlink 1509 (time 1787134678.04 total 10.00 last 150.88) - /unlink 2809 (time 1787134688.05 total 20.00 last 129.97) - /unlink 4219 (time 1787134698.06 total 30.02 last 140.79) total: 4500 /unlink in 32.16 seconds: 139.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 777 (time 1787134715.41 total 10.00 last 77.67) - open/close 1680 (time 1787134725.43 total 20.02 last 90.17) - open/close 2571 (time 1787134735.43 total 30.02 last 89.08) - open/close 3462 (time 1787134745.44 total 40.03 last 89.05) - open/close 4240 (time 1787134755.45 total 50.04 last 77.73) total: 4500 open/close in 53.67 seconds: 83.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1616 (time 1787134773.28 total 10.00 last 161.54) - /unlink 3314 (time 1787134783.28 total 20.00 last 169.79) total: 4500 /unlink in 27.03 seconds: 166.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 888 (time 1787134804.67 total 10.00 last 88.76) - open/close 1811 (time 1787134814.67 total 20.01 last 92.29) - open/close 2762 (time 1787134824.68 total 30.01 last 95.08) - open/close 3582 (time 1787134834.68 total 40.01 last 81.95) - open/close 4393 (time 1787134844.69 total 50.02 last 81.02) total: 4500 open/close in 51.39 seconds: 87.56 ops/second - /unlink 1488 (time 1787134860.46 total 10.01 last 148.68) - /unlink 2987 (time 1787134870.46 total 20.01 last 149.90) - /unlink 4491 (time 1787134880.47 total 30.01 last 150.37) total: 4500 /unlink in 30.07 seconds: 149.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 864 (time 1787134895.33 total 10.02 last 86.20) - open/close 1572 (time 1787134905.34 total 20.03 last 70.75) - open/close 2403 (time 1787134915.34 total 30.04 last 83.04) - open/close 3338 (time 1787134925.35 total 40.04 last 93.46) - open/close 4169 (time 1787134935.35 total 50.04 last 83.08) total: 4500 open/close in 53.72 seconds: 83.77 ops/second - /unlink 1396 (time 1787134953.18 total 10.00 last 139.58) - /unlink 2976 (time 1787134963.19 total 20.01 last 157.89) total: 4500 /unlink in 29.85 seconds: 150.76 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95293 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server unloading modules via unload_modules_local on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local oleg136-server: modules unloaded.