Now run the local parser, parser.py via the terminal. The RegEx capturing group for the timestamp field must be named timestamp in the RegEx provided. There must be a RegEx capturing group named uri which captures the uri of the logged event. The RegEx must be in the Python format.
1 2 3 4 5 6 7
:param file: Name of log file to parse. Accepts wildcards. :param app: Name of the application for the logs. :param bucket: S3 bucket to store the parsed logs to, Ex: "my-bucket123" :param timeformat: The format of the timestamp in the logs. Ex: 'DD/MMM/YYYY:HH:mm:ss ZZ' Accepts the following tokens: https://pendulum.eustace.io/docs/#tokens :param regex: Regex to use to parse the logs. Ex: '(?P<remote_addr>[\S]+) - (?P<remote_user>[\S]+) \[(?P<timestamp>.+)\] "(?P<req_method>.+) (?P<uri>.+) (?P<httpver>.+)" (?P<status>[\S]+) (?P<body_bytes_sent>[\S]+) "(?P<referer>[\S]+)" "(?P<user_agent>[\S]+)" "(?P<x_forwarded_for>[\S]+)"'
Run the local parser
1 2
# inside the shadowreader directory pip install -r requirements-local-parser.txt
NOTE: The S3 bucket set in --bucket must be the same as the name of the deployed parsed_data_bucket in serverless.yml
You should see an output like below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
5 minutes of traffic data was uploaded to S3. Average requests/min: 6 Max requests/min: 8 Min requests/min: 2 Timezone found in logs: +00:00 To load test with these results, use the below parameters for the orchestrator in serverless.yml ========================================== test_params: { "base_url": "http://$your_base_url", "rate": 100, "replay_start_time": "2019-03-15T04:12", "replay_end_time": "2019-03-15T04:16", "identifier": "oss" } apps_to_test: ["app1"] ==========================================
Paste the test_params and apps_to_test into serverless.yml and follow the other guides to start the load test.
