https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
https://reiki4040.hatenablog.com/entry/2020/02/03/083000
CloudWatch Logs ---> Kinesis Data Firehose ---> S3
-- 1. コマンド等のインストール
-- 1.1 aws cli version 2 インストール
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version
-- 1.2 jqインストール
sudo yum -y install jq
-- 2. CloudWatchロググループの作成
aws logs create-log-group --log-group-name app01/test.log
aws logs describe-log-groups --log-group-name-prefix app01/test.log
aws logs put-retention-policy \
--log-group-name app01/test.log \
--retention-in-days 1
-- 3. CloudWatchログストリームの作成
aws logs create-log-stream \
--log-group-name app01/test.log \
--log-stream-name stream01
aws logs describe-log-streams \
--log-group-name app01/test.log
-- 4. S3 バケットを作成する
aws s3 mb s3://bucket123
aws s3 ls
-- 5. IAMポリシー作成(Amazon S3 バケットにデータを置く権限を Amazon Kinesis Data Firehose に付与)
vim policy01.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutObject" ],
"Resource": [
"arn:aws:s3:::bucket123",
"arn:aws:s3:::bucket123/*" ]
}
]
}
aws iam create-policy \
--policy-name policy01 \
--policy-document file://policy01.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "firehose.amazonaws.com" },
"Action": "sts:AssumeRole"
}
]
}
aws iam create-role \
--role-name role01 \
--assume-role-policy-document file://role01.json
-- 7. ポリシーをロールにアタッチ
aws iam attach-role-policy \
--policy-arn arn:aws:iam::999999999999:policy/policy01 \
--role-name role01
-- 8. Kinesis Data Firehose 送信ストリームを作成
aws firehose create-delivery-stream \
--delivery-stream-name firehose01 \
--s3-destination-configuration '{"RoleARN": "arn:aws:iam::999999999999:role/role01", "BucketARN": "arn:aws:s3:::bucket123"}'
aws firehose describe-delivery-stream \
--delivery-stream-name firehose01
-- 9. IAMポリシー作成(Kinesis Data Firehose 送信ストリームにデータを置く権限を CloudWatch Logs に付与)
vim policy02.json
{
"Version": "2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":["firehose:*"],
"Resource":["arn:aws:firehose:ap-northeast-1:999999999999:*"]
}
]
}
aws iam create-policy \
--policy-name policy02 \
--policy-document file://policy02.json
-- 10. IAMロール作成
vim role02.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "logs.ap-northeast-1.amazonaws.com" },
"Action": "sts:AssumeRole",
"Condition": {
"StringLike": { "aws:SourceArn": "arn:aws:logs:ap-northeast-1:999999999999:*" }
}
}
]
}
aws iam create-role \
--role-name role02 \
--assume-role-policy-document file://role02.json
-- 11. ポリシーをロールにアタッチ
aws iam attach-role-policy \
--policy-arn arn:aws:iam::999999999999:policy/policy02 \
--role-name role02
-- 12. サブスクリプションフィルタ作成
aws logs put-subscription-filter \
--log-group-name "app01/test.log" \
--filter-name "filter01" \
--filter-pattern "Error" \
--destination-arn "arn:aws:firehose:ap-northeast-1:999999999999:deliverystream/firehose01" \
--role-arn "arn:aws:iam::999999999999:role/role02"
-- 13. 動作確認
aws logs put-log-events \
--log-group-name app01/test.log \
--log-stream-name stream01 \
--log-events \
timestamp=1646547311000,message="Error01" \
timestamp=1646547322000,message="OK01" \
timestamp=1646547333000,message="Error02"
Amazon Kinesis Data Firehose 配信ストリームに設定された時間バッファ間隔に基づいて、Amazon S3 にデータが表示されるようになります。
Buffer interval = 300 seconds
なので、しばらく待つ。
aws s3 ls s3://bucket123 --recursive
aws s3 cp s3://bucket123/2022/03/06/10/firehose01-1-2022-03-06-10-44-48-12643c0d-f8c4-4560-bdf3-340f4ee8e75e - | zcat
"OK01"はS3に届いていないことを確認
-- 14. クリーンアップ
-- サブスクリプションフィルターの削除
aws logs describe-subscription-filters \
--log-group-name app01/test.log
aws logs delete-subscription-filter \
--log-group-name "app01/test.log" \
--filter-name "filter01"
-- Kinesis Data Firehose 送信ストリームを削除
aws firehose describe-delivery-stream \
--delivery-stream-name firehose01
aws firehose delete-delivery-stream \
--delivery-stream-name firehose01
-- IAMロールの削除
aws iam list-roles | grep role01
aws iam list-roles | grep role02
aws iam detach-role-policy \
--role-name role01 \
--policy-arn arn:aws:iam::999999999999:policy/policy01
aws iam detach-role-policy \
--role-name role02 \
--policy-arn arn:aws:iam::999999999999:policy/policy02
aws iam delete-role --role-name role01
aws iam delete-role --role-name role02
-- IAMポリシーの削除
aws iam list-policies | grep policy01
aws iam list-policies | grep policy02
aws iam delete-policy \
--policy-arn arn:aws:iam::999999999999:policy/policy01
aws iam delete-policy \
--policy-arn arn:aws:iam::999999999999:policy/policy02
-- S3バケットの削除
aws s3 ls
aws s3 rb s3://bucket123 --force
-- CloudWatchログストリームの削除
aws logs describe-log-streams \
--log-group-name app01/test.log
aws logs delete-log-stream \
--log-group-name app01/test.log \
--log-stream-name stream01
-- CloudWatchロググループの削除
aws logs describe-log-groups --log-group-name-prefix app01/test.log
aws logs delete-log-group --log-group-name app01/test.log
