== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:34:45 (1787834085) Stopping clients: oleg645-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg645-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg645-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg645-server oleg645-server: oleg645-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg645-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg645-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 737 (time 1787834278.95 total 10.01 last 73.63) - open/close 1362 (time 1787834288.96 total 20.02 last 62.43) - open/close 2087 (time 1787834298.98 total 30.04 last 72.38) - open/close 2856 (time 1787834308.99 total 40.05 last 76.84) - open/close 3497 (time 1787834319.01 total 50.07 last 63.97) - open/close 4153 (time 1787834329.01 total 60.07 last 65.57) total: 4500 open/close in 64.65 seconds: 69.61 ops/second - /unlink 1397 (time 1787834348.34 total 10.00 last 139.65) - /unlink 2847 (time 1787834358.34 total 20.01 last 144.95) - /unlink 4296 (time 1787834368.34 total 30.01 last 144.86) total: 4500 /unlink in 31.33 seconds: 143.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 805 (time 1787834383.62 total 10.00 last 80.48) - open/close 1586 (time 1787834393.64 total 20.02 last 77.95) - open/close 2366 (time 1787834403.64 total 30.03 last 77.95) - open/close 3212 (time 1787834413.65 total 40.04 last 84.53) - open/close 3997 (time 1787834423.66 total 50.05 last 78.43) total: 4500 open/close in 56.28 seconds: 79.96 ops/second - /unlink 1278 (time 1787834445.31 total 10.00 last 127.76) - /unlink 2612 (time 1787834455.32 total 20.01 last 133.34) - /unlink 3766 (time 1787834465.32 total 30.01 last 115.35) total: 4500 /unlink in 35.65 seconds: 126.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 789 (time 1787834485.28 total 10.01 last 78.86) - open/close 1590 (time 1787834495.29 total 20.01 last 80.02) - open/close 2370 (time 1787834505.30 total 30.03 last 77.89) - open/close 3155 (time 1787834515.31 total 40.03 last 78.47) - open/close 3925 (time 1787834525.32 total 50.04 last 76.93) total: 4500 open/close in 57.58 seconds: 78.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1507 (time 1787834546.64 total 10.01 last 150.49) - /unlink 2952 (time 1787834556.64 total 20.01 last 144.49) - /unlink 4382 (time 1787834566.65 total 30.02 last 142.90) total: 4500 /unlink in 30.93 seconds: 145.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 746 (time 1787834581.36 total 10.01 last 74.53) - open/close 1488 (time 1787834591.36 total 20.01 last 74.20) - open/close 2297 (time 1787834601.37 total 30.01 last 80.87) - open/close 3044 (time 1787834611.38 total 40.03 last 74.59) - open/close 3800 (time 1787834621.39 total 50.04 last 75.55) total: 4500 open/close in 58.88 seconds: 76.43 ops/second - /unlink 1445 (time 1787834644.01 total 10.00 last 144.44) - /unlink 2806 (time 1787834654.02 total 20.01 last 136.05) - /unlink 4155 (time 1787834664.02 total 30.01 last 134.86) total: 4500 /unlink in 32.14 seconds: 140.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 791 (time 1787834681.10 total 10.00 last 79.10) - open/close 1521 (time 1787834691.11 total 20.01 last 72.95) - open/close 2239 (time 1787834701.13 total 30.03 last 71.65) - open/close 2963 (time 1787834711.15 total 40.05 last 72.24) - open/close 3710 (time 1787834721.15 total 50.05 last 74.67) - open/close 4482 (time 1787834731.16 total 60.06 last 77.15) total: 4500 open/close in 60.28 seconds: 74.65 ops/second - /unlink 1262 (time 1787834745.69 total 10.01 last 126.08) - /unlink 2599 (time 1787834755.70 total 20.02 last 133.61) - /unlink 3935 (time 1787834765.70 total 30.02 last 133.57) total: 4500 /unlink in 34.26 seconds: 131.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 707 (time 1787834784.71 total 10.01 last 70.64) - open/close 1418 (time 1787834794.71 total 20.01 last 71.07) - open/close 2179 (time 1787834804.73 total 30.03 last 75.95) - open/close 2895 (time 1787834814.75 total 40.04 last 71.52) - open/close 3596 (time 1787834824.75 total 50.05 last 70.08) - open/close 4280 (time 1787834834.75 total 60.05 last 68.38) total: 4500 open/close in 63.05 seconds: 71.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1433 (time 1787834852.15 total 10.00 last 143.28) - /unlink 2865 (time 1787834862.15 total 20.00 last 143.16) - /unlink 4356 (time 1787834872.15 total 30.01 last 149.08) total: 4500 /unlink in 31.12 seconds: 144.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 767 (time 1787834887.40 total 10.01 last 76.66) - open/close 1575 (time 1787834897.40 total 20.01 last 80.78) - open/close 2368 (time 1787834907.42 total 30.02 last 79.19) - open/close 3152 (time 1787834917.42 total 40.03 last 78.36) - open/close 3983 (time 1787834927.43 total 50.03 last 83.06) total: 4500 open/close in 56.47 seconds: 79.69 ops/second - /unlink 1499 (time 1787834947.42 total 10.00 last 149.89) - /unlink 3010 (time 1787834957.42 total 20.01 last 150.99) total: 4500 /unlink in 29.93 seconds: 150.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 779 (time 1787834981.04 total 10.00 last 77.87) - open/close 1506 (time 1787834991.05 total 20.01 last 72.67) - open/close 2251 (time 1787835001.05 total 30.01 last 74.48) - open/close 2988 (time 1787835011.05 total 40.01 last 73.68) - open/close 3708 (time 1787835021.06 total 50.02 last 71.95) - open/close 4463 (time 1787835031.07 total 60.03 last 75.45) total: 4500 open/close in 60.49 seconds: 74.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1432 (time 1787835046.06 total 10.00 last 143.20) - /unlink 2958 (time 1787835056.06 total 20.01 last 152.50) - /unlink 4239 (time 1787835066.07 total 30.01 last 128.02) total: 4500 /unlink in 32.08 seconds: 140.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 738 (time 1787835083.11 total 10.01 last 73.74) - open/close 1440 (time 1787835093.12 total 20.02 last 70.13) - open/close 2204 (time 1787835103.13 total 30.03 last 76.31) - open/close 2942 (time 1787835113.14 total 40.03 last 73.78) - open/close 3601 (time 1787835123.16 total 50.05 last 65.76) - open/close 4352 (time 1787835133.16 total 60.05 last 75.10) total: 4500 open/close in 62.51 seconds: 71.99 ops/second - /unlink 1120 (time 1787835149.63 total 10.00 last 111.95) - /unlink 2068 (time 1787835159.63 total 20.01 last 94.77) - /unlink 3271 (time 1787835169.63 total 30.01 last 120.30) total: 4500 /unlink in 39.52 seconds: 113.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 680 (time 1787835193.41 total 10.01 last 67.94) - open/close 1397 (time 1787835203.42 total 20.02 last 71.63) - open/close 2106 (time 1787835213.42 total 30.02 last 70.87) - open/close 2814 (time 1787835223.43 total 40.03 last 70.75) - open/close 3570 (time 1787835233.44 total 50.04 last 75.54) - open/close 4330 (time 1787835243.44 total 60.04 last 75.98) total: 4500 open/close in 62.43 seconds: 72.09 ops/second - /unlink 1478 (time 1787835259.86 total 10.00 last 147.75) - /unlink 2891 (time 1787835269.86 total 20.01 last 141.24) - /unlink 4375 (time 1787835279.87 total 30.01 last 148.35) total: 4500 /unlink in 31.00 seconds: 145.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 721 (time 1787835294.53 total 10.00 last 72.09) - open/close 1433 (time 1787835304.55 total 20.02 last 71.10) - open/close 2182 (time 1787835314.55 total 30.02 last 74.89) - open/close 2973 (time 1787835324.55 total 40.02 last 79.09) - open/close 3759 (time 1787835334.55 total 50.02 last 78.58) - open/close 4491 (time 1787835344.56 total 60.03 last 73.16) total: 4500 open/close in 60.13 seconds: 74.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1203 (time 1787835358.80 total 10.01 last 120.19) - /unlink 2207 (time 1787835368.81 total 20.02 last 100.24) - /unlink 3524 (time 1787835378.81 total 30.03 last 131.67) total: 4500 /unlink in 36.83 seconds: 122.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 738 (time 1787835399.19 total 10.02 last 73.64) - open/close 1374 (time 1787835409.20 total 20.04 last 63.50) - open/close 1887 (time 1787835419.22 total 30.05 last 51.23) - open/close 2417 (time 1787835429.23 total 40.07 last 52.93) - open/close 3036 (time 1787835439.23 total 50.07 last 61.89) - open/close 3722 (time 1787835449.25 total 60.08 last 68.50) - open/close 4474 (time 1787835459.25 total 70.08 last 75.19) total: 4500 open/close in 70.40 seconds: 63.92 ops/second - /unlink 1353 (time 1787835473.49 total 10.01 last 135.20) - /unlink 2772 (time 1787835483.49 total 20.01 last 141.81) - /unlink 4238 (time 1787835493.50 total 30.02 last 146.53) total: 4500 /unlink in 31.93 seconds: 140.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 806 (time 1787835508.90 total 10.01 last 80.49) - open/close 1552 (time 1787835518.91 total 20.02 last 74.54) - open/close 2305 (time 1787835528.91 total 30.03 last 75.24) - open/close 3070 (time 1787835538.92 total 40.04 last 76.44) - open/close 3825 (time 1787835548.93 total 50.05 last 75.44) total: 4500 open/close in 59.83 seconds: 75.21 ops/second - /unlink 1366 (time 1787835572.76 total 10.00 last 136.59) - /unlink 2652 (time 1787835582.77 total 20.00 last 128.54) - /unlink 3889 (time 1787835592.77 total 30.01 last 123.61) total: 4500 /unlink in 34.23 seconds: 131.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 714 (time 1787835611.38 total 10.01 last 71.34) - open/close 1389 (time 1787835621.39 total 20.01 last 67.46) - open/close 2118 (time 1787835631.39 total 30.02 last 72.85) - open/close 2857 (time 1787835641.40 total 40.03 last 73.85) - open/close 3637 (time 1787835651.41 total 50.03 last 77.95) - open/close 4381 (time 1787835661.43 total 60.06 last 74.23) total: 4500 open/close in 62.11 seconds: 72.45 ops/second - /unlink 1260 (time 1787835678.35 total 10.00 last 126.00) - /unlink 2572 (time 1787835688.35 total 20.00 last 131.16) - /unlink 3843 (time 1787835698.35 total 30.01 last 127.06) total: 4500 /unlink in 35.37 seconds: 127.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 630 (time 1787835718.11 total 10.00 last 62.98) - open/close 1186 (time 1787835728.13 total 20.02 last 55.48) - open/close 1703 (time 1787835738.13 total 30.03 last 51.70) - open/close 2433 (time 1787835748.14 total 40.03 last 72.96) - open/close 3160 (time 1787835758.15 total 50.05 last 72.60) - open/close 3925 (time 1787835768.16 total 60.05 last 76.45) total: 4500 open/close in 67.32 seconds: 66.85 ops/second - /unlink 1331 (time 1787835790.07 total 10.00 last 133.06) - /unlink 2457 (time 1787835800.08 total 20.01 last 112.52) - /unlink 3858 (time 1787835810.08 total 30.01 last 140.09) total: 4500 /unlink in 34.44 seconds: 130.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 721 (time 1787835828.60 total 10.00 last 72.09) - open/close 1439 (time 1787835838.60 total 20.00 last 71.79) - open/close 2049 (time 1787835848.60 total 30.00 last 60.99) - open/close 2688 (time 1787835858.63 total 40.03 last 63.71) - open/close 3410 (time 1787835868.64 total 50.04 last 72.14) - open/close 4186 (time 1787835878.65 total 60.06 last 77.48) total: 4500 open/close in 64.06 seconds: 70.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1425 (time 1787835896.88 total 10.00 last 142.47) - /unlink 2811 (time 1787835906.89 total 20.01 last 138.51) - /unlink 4181 (time 1787835916.89 total 30.01 last 137.00) total: 4500 /unlink in 32.25 seconds: 139.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 677 (time 1787835934.00 total 10.01 last 67.64) - open/close 1387 (time 1787835944.01 total 20.02 last 70.93) - open/close 2172 (time 1787835954.02 total 30.03 last 78.44) - open/close 2900 (time 1787835964.02 total 40.03 last 72.76) - open/close 3629 (time 1787835974.03 total 50.04 last 72.85) - open/close 4168 (time 1787835984.05 total 60.06 last 53.79) total: 4500 open/close in 65.04 seconds: 69.18 ops/second - /unlink 1384 (time 1787836004.26 total 10.01 last 138.22) - /unlink 2601 (time 1787836014.26 total 20.01 last 121.68) - /unlink 3877 (time 1787836024.27 total 30.02 last 127.51) total: 4500 /unlink in 35.02 seconds: 128.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 710 (time 1787836043.27 total 10.01 last 70.91) - open/close 1445 (time 1787836053.28 total 20.01 last 73.49) - open/close 2217 (time 1787836063.28 total 30.02 last 77.15) - open/close 2879 (time 1787836073.29 total 40.03 last 66.15) - open/close 3542 (time 1787836083.29 total 50.03 last 66.28) - open/close 4294 (time 1787836093.31 total 60.05 last 75.08) total: 4500 open/close in 62.89 seconds: 71.56 ops/second - /unlink 1346 (time 1787836109.87 total 10.01 last 134.44) - /unlink 2589 (time 1787836119.87 total 20.01 last 124.29) - /unlink 3811 (time 1787836129.87 total 30.01 last 122.18) total: 4500 /unlink in 36.09 seconds: 124.70 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95177 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg645-server unloading modules via unload_modules_local on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing unload_modules_local oleg645-server: modules unloaded.