== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:33:00 (1787473980) Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg224-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg224-server oleg224-server: oleg224-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg224-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg224-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg224-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 663 (time 1787474205.93 total 10.02 last 66.19) - open/close 1375 (time 1787474215.95 total 20.03 last 71.10) - open/close 2069 (time 1787474225.96 total 30.04 last 69.32) - open/close 2760 (time 1787474235.98 total 40.07 last 68.93) - open/close 3438 (time 1787474245.99 total 50.08 last 67.74) - open/close 4085 (time 1787474256.01 total 60.09 last 64.58) total: 4500 open/close in 65.55 seconds: 68.65 ops/second - /unlink 1317 (time 1787474276.16 total 10.01 last 131.57) - /unlink 2395 (time 1787474286.17 total 20.02 last 107.72) - /unlink 3800 (time 1787474296.17 total 30.02 last 140.47) total: 4500 /unlink in 35.87 seconds: 125.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 489 (time 1787474317.21 total 10.01 last 48.86) - open/close 1000 (time 1787474327.23 total 20.03 last 50.99) - open/close 1569 (time 1787474337.25 total 30.05 last 56.78) - open/close 2142 (time 1787474347.26 total 40.05 last 57.29) - open/close 2717 (time 1787474357.26 total 50.06 last 57.46) - open/close 3421 (time 1787474367.28 total 60.08 last 70.26) - open/close 4178 (time 1787474377.28 total 70.08 last 75.68) total: 4500 open/close in 75.42 seconds: 59.67 ops/second - /unlink 1192 (time 1787474398.02 total 10.00 last 119.18) - /unlink 2347 (time 1787474408.02 total 20.00 last 115.50) - /unlink 3778 (time 1787474418.02 total 30.00 last 143.09) total: 4500 /unlink in 34.94 seconds: 128.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 847 (time 1787474437.20 total 10.01 last 84.62) - open/close 1569 (time 1787474447.22 total 20.03 last 72.04) - open/close 2298 (time 1787474457.24 total 30.06 last 72.72) - open/close 3023 (time 1787474467.28 total 40.09 last 72.22) - open/close 3774 (time 1787474477.29 total 50.10 last 75.05) total: 4500 open/close in 58.99 seconds: 76.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1318 (time 1787474500.65 total 10.00 last 131.79) - /unlink 2606 (time 1787474510.66 total 20.01 last 128.72) - /unlink 3958 (time 1787474520.67 total 30.02 last 135.05) total: 4500 /unlink in 34.05 seconds: 132.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 713 (time 1787474540.50 total 10.00 last 71.29) - open/close 1476 (time 1787474550.51 total 20.01 last 76.25) - open/close 2252 (time 1787474560.51 total 30.01 last 77.58) - open/close 2974 (time 1787474570.52 total 40.02 last 72.16) - open/close 3698 (time 1787474580.52 total 50.03 last 72.34) total: 4500 open/close in 60.02 seconds: 74.97 ops/second - /unlink 1460 (time 1787474605.53 total 10.00 last 145.94) - /unlink 2785 (time 1787474615.53 total 20.01 last 132.47) - /unlink 4219 (time 1787474625.54 total 30.01 last 143.33) total: 4500 /unlink in 32.44 seconds: 138.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 707 (time 1787474643.47 total 10.02 last 70.59) - open/close 1611 (time 1787474653.48 total 20.02 last 90.32) - open/close 2369 (time 1787474663.49 total 30.04 last 75.72) - open/close 3056 (time 1787474673.51 total 40.05 last 68.57) - open/close 3758 (time 1787474683.52 total 50.06 last 70.16) - open/close 4489 (time 1787474693.52 total 60.07 last 73.05) total: 4500 open/close in 60.23 seconds: 74.71 ops/second - /unlink 1237 (time 1787474709.56 total 10.00 last 123.67) - /unlink 2612 (time 1787474719.57 total 20.01 last 137.42) - /unlink 3752 (time 1787474729.57 total 30.02 last 113.91) total: 4500 /unlink in 36.37 seconds: 123.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 537 (time 1787474751.41 total 10.01 last 53.66) - open/close 1194 (time 1787474761.42 total 20.02 last 65.60) - open/close 1787 (time 1787474771.43 total 30.03 last 59.25) - open/close 2587 (time 1787474781.44 total 40.04 last 79.90) - open/close 3298 (time 1787474791.45 total 50.05 last 71.07) - open/close 3969 (time 1787474801.46 total 60.06 last 66.99) total: 4500 open/close in 67.17 seconds: 66.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1413 (time 1787474823.23 total 10.00 last 141.26) - /unlink 2792 (time 1787474833.24 total 20.01 last 137.82) - /unlink 4051 (time 1787474843.24 total 30.01 last 125.85) total: 4500 /unlink in 34.72 seconds: 129.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 774 (time 1787474862.73 total 10.01 last 77.36) - open/close 1579 (time 1787474872.74 total 20.01 last 80.48) - open/close 2238 (time 1787474882.75 total 30.02 last 65.79) - open/close 2950 (time 1787474892.76 total 40.03 last 71.18) - open/close 3721 (time 1787474902.76 total 50.03 last 77.07) - open/close 4471 (time 1787474912.76 total 60.03 last 75.00) total: 4500 open/close in 60.76 seconds: 74.06 ops/second - /unlink 1138 (time 1787474928.89 total 10.00 last 113.79) - /unlink 2433 (time 1787474938.89 total 20.00 last 129.50) - /unlink 3737 (time 1787474948.90 total 30.01 last 130.28) total: 4500 /unlink in 37.87 seconds: 118.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 765 (time 1787474972.11 total 10.00 last 76.48) - open/close 1436 (time 1787474982.12 total 20.01 last 67.02) - open/close 2106 (time 1787474992.12 total 30.02 last 66.96) - open/close 2717 (time 1787475002.14 total 40.04 last 61.00) - open/close 3166 (time 1787475012.16 total 50.06 last 44.81) - open/close 3916 (time 1787475022.17 total 60.06 last 74.95) total: 4500 open/close in 68.89 seconds: 65.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1369 (time 1787475047.19 total 10.00 last 136.88) - /unlink 2561 (time 1787475057.20 total 20.01 last 119.15) - /unlink 3756 (time 1787475067.20 total 30.01 last 119.45) total: 4500 /unlink in 34.75 seconds: 129.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 556 (time 1787475087.54 total 10.01 last 55.55) - open/close 1167 (time 1787475097.54 total 20.01 last 61.09) - open/close 1872 (time 1787475107.55 total 30.02 last 70.44) - open/close 2617 (time 1787475117.56 total 40.03 last 74.43) - open/close 3381 (time 1787475127.56 total 50.03 last 76.35) - open/close 4099 (time 1787475137.58 total 60.05 last 71.66) total: 4500 open/close in 65.77 seconds: 68.42 ops/second - /unlink 1315 (time 1787475157.93 total 10.00 last 131.45) - /unlink 2483 (time 1787475167.93 total 20.01 last 116.70) - /unlink 3654 (time 1787475177.94 total 30.01 last 117.09) total: 4500 /unlink in 38.09 seconds: 118.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 508 (time 1787475203.11 total 10.02 last 50.71) - open/close 948 (time 1787475213.14 total 20.05 last 43.86) - open/close 1619 (time 1787475223.17 total 30.07 last 66.94) - open/close 2295 (time 1787475233.20 total 40.11 last 67.35) - open/close 2812 (time 1787475243.21 total 50.12 last 51.65) - open/close 3531 (time 1787475253.22 total 60.12 last 71.87) - open/close 4143 (time 1787475263.22 total 70.12 last 61.19) total: 4500 open/close in 76.35 seconds: 58.94 ops/second - /unlink 1496 (time 1787475284.10 total 10.00 last 149.55) - /unlink 2804 (time 1787475294.10 total 20.00 last 130.80) - /unlink 4038 (time 1787475304.11 total 30.01 last 123.37) total: 4500 /unlink in 34.21 seconds: 131.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 606 (time 1787475324.03 total 10.01 last 60.56) - open/close 1191 (time 1787475334.04 total 20.02 last 58.45) - open/close 1788 (time 1787475344.04 total 30.02 last 59.69) - open/close 2449 (time 1787475354.06 total 40.03 last 66.00) - open/close 3203 (time 1787475364.06 total 50.04 last 75.33) - open/close 3970 (time 1787475374.07 total 60.04 last 76.69) total: 4500 open/close in 66.89 seconds: 67.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1335 (time 1787475396.11 total 10.01 last 133.30) - /unlink 2573 (time 1787475406.11 total 20.02 last 123.76) - /unlink 3692 (time 1787475416.11 total 30.02 last 111.90) total: 4500 /unlink in 35.54 seconds: 126.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 637 (time 1787475438.55 total 10.00 last 63.67) - open/close 1270 (time 1787475448.56 total 20.02 last 63.22) - open/close 1873 (time 1787475458.59 total 30.05 last 60.12) - open/close 2364 (time 1787475468.60 total 40.05 last 49.08) - open/close 3044 (time 1787475478.61 total 50.06 last 67.92) - open/close 3730 (time 1787475488.64 total 60.09 last 68.40) - open/close 4345 (time 1787475498.64 total 70.09 last 61.49) total: 4500 open/close in 71.84 seconds: 62.64 ops/second - /unlink 1355 (time 1787475515.75 total 10.00 last 135.44) - /unlink 2390 (time 1787475525.76 total 20.01 last 103.44) - /unlink 3664 (time 1787475535.78 total 30.03 last 127.13) total: 4500 /unlink in 37.18 seconds: 121.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 776 (time 1787475557.98 total 10.02 last 77.45) - open/close 1542 (time 1787475567.99 total 20.03 last 76.53) - open/close 2254 (time 1787475577.99 total 30.03 last 71.19) - open/close 3056 (time 1787475588.00 total 40.04 last 80.12) - open/close 3831 (time 1787475598.00 total 50.04 last 77.47) total: 4500 open/close in 58.44 seconds: 77.00 ops/second - /unlink 1489 (time 1787475621.23 total 10.00 last 148.86) - /unlink 2884 (time 1787475631.23 total 20.01 last 139.44) - /unlink 4270 (time 1787475641.24 total 30.01 last 138.50) total: 4500 /unlink in 32.00 seconds: 140.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 729 (time 1787475657.93 total 10.00 last 72.90) - open/close 1357 (time 1787475667.93 total 20.00 last 62.77) - open/close 1950 (time 1787475677.94 total 30.01 last 59.26) - open/close 2781 (time 1787475687.94 total 40.01 last 83.10) - open/close 3466 (time 1787475697.95 total 50.02 last 68.46) - open/close 3999 (time 1787475707.96 total 60.03 last 53.23) total: 4500 open/close in 69.80 seconds: 64.47 ops/second - /unlink 1092 (time 1787475733.13 total 10.00 last 109.16) - /unlink 2355 (time 1787475743.13 total 20.01 last 126.25) - /unlink 3943 (time 1787475753.14 total 30.01 last 158.74) total: 4500 /unlink in 34.19 seconds: 131.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 785 (time 1787475771.64 total 10.00 last 78.49) - open/close 1530 (time 1787475781.64 total 20.00 last 74.48) - open/close 2278 (time 1787475791.65 total 30.01 last 74.74) - open/close 3035 (time 1787475801.67 total 40.03 last 75.58) - open/close 3798 (time 1787475811.67 total 50.03 last 76.29) total: 4500 open/close in 59.12 seconds: 76.12 ops/second - /unlink 1311 (time 1787475835.70 total 10.01 last 130.92) - /unlink 2572 (time 1787475845.71 total 20.02 last 125.97) - /unlink 3928 (time 1787475855.71 total 30.02 last 135.59) total: 4500 /unlink in 34.51 seconds: 130.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 731 (time 1787475874.99 total 10.03 last 72.91) - open/close 1273 (time 1787475885.01 total 20.05 last 54.08) - open/close 1798 (time 1787475895.02 total 30.05 last 52.47) - open/close 2625 (time 1787475905.02 total 40.06 last 82.69) - open/close 3410 (time 1787475915.02 total 50.06 last 78.49) - open/close 4157 (time 1787475925.03 total 60.06 last 74.65) total: 4500 open/close in 64.46 seconds: 69.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1378 (time 1787475943.97 total 10.01 last 137.69) - /unlink 2738 (time 1787475953.97 total 20.01 last 135.91) - /unlink 4047 (time 1787475963.98 total 30.02 last 130.86) total: 4500 /unlink in 33.63 seconds: 133.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 692 (time 1787475982.68 total 10.01 last 69.14) - open/close 1219 (time 1787475992.68 total 20.01 last 52.68) - open/close 1884 (time 1787476002.69 total 30.02 last 66.45) - open/close 2610 (time 1787476012.69 total 40.02 last 72.59) - open/close 3433 (time 1787476022.69 total 50.03 last 82.27) - open/close 4170 (time 1787476032.71 total 60.04 last 73.62) total: 4500 open/close in 64.42 seconds: 69.86 ops/second - /unlink 1448 (time 1787476052.25 total 10.00 last 144.73) - /unlink 2664 (time 1787476062.26 total 20.01 last 121.56) - /unlink 4111 (time 1787476072.26 total 30.01 last 144.68) total: 4500 /unlink in 33.24 seconds: 135.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 593 (time 1787476091.96 total 10.00 last 59.28) - open/close 1159 (time 1787476101.98 total 20.02 last 56.52) - open/close 1744 (time 1787476111.98 total 30.02 last 58.47) - open/close 2376 (time 1787476121.98 total 40.03 last 63.18) - open/close 3160 (time 1787476131.99 total 50.03 last 78.33) - open/close 3887 (time 1787476142.00 total 60.04 last 72.65) total: 4500 open/close in 67.54 seconds: 66.63 ops/second - /unlink 1295 (time 1787476164.63 total 10.00 last 129.47) - /unlink 2524 (time 1787476174.63 total 20.01 last 122.85) - /unlink 3680 (time 1787476184.64 total 30.01 last 115.54) total: 4500 /unlink in 37.54 seconds: 119.88 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95321 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: modules unloaded.