README
¶
Dynamomigration tool
Dynamomigration tool allows to export Teleport audit events logs from DynamoDB table into Athena Audit log. It's using DynamoDB export to S3 to export data.
Requirements:
- Point-in-time recovery (PITR) on DynamoDB table
- Writable filesystem on machine where script will be executed
- IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowDynamoExportAndList",
"Effect": "Allow",
"Action": [
"dynamodb:ExportTableToPointInTime"
],
"Resource": "arn:aws:dynamodb:region:account:table/tablename"
},
{
"Sid": "AllowDynamoExportDescribe",
"Effect": "Allow",
"Action": [
"dynamodb:DescribeExport"
],
"Resource": "arn:aws:dynamodb:region:account:table/tablename/*"
},
{
"Sid": "AllowWriteReadDestinationBucket",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObject"
],
"Resource": "arn:aws:s3:::export-bucket/*"
},
{
"Sid": "AllowWriteLargePayloadsBucket",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::large-payloads-bucket/*"
},
{
"Sid": "AllowPublishToAthenaTopic",
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": "arn:aws:sns:region:account:topicname"
}
]
}
Example usage
Build: cd examples/dynamoathenamigration/cmd && go build -o dynamoathenamigration.
It is recommended to test export first using -dryRun flag. DryRun does not emit any events,
it makes sure that export is in valid format and events can be parsed.
Dry run example:
./dynamoathenamigration -dynamoARN='arn:aws:dynamodb:region:account:table/tablename' \
-exportPath='s3://bucket/prefix' \
-dryRun
Full migration:
./dynamoathenamigration -dynamoARN='arn:aws:dynamodb:region:account:table/tablename' \
-exportPath='s3://bucket/prefix' \
-snsTopicARN=arn:aws:sns:region:account:topicname \
-largePayloadsPath=s3://bucket/prefix
To reuse existing export without triggering new one, use -exportARN=xxx.
Documentation
¶
Index ¶
Constants ¶
View Source
const (
DefaultMaxMemoryUsedForSortingExportInMB = 500
)
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Config ¶
type Config struct {
// ExportTime is time in the past from which to export table data.
ExportTime time.Time
// ExportARN allows to use already finished export without triggering new.
ExportARN string
// DynamoTableARN that will be exported.
DynamoTableARN string
// ExportLocalDir specifies where export files will be downloaded (it must exists).
// If empty os.TempDir() will be used.
ExportLocalDir string
// MaxMemoryUsedForSortingExportInMB (MB) is used to define how large amount of events
// will be loaded into memory when doing sorting of events before publishing it.
MaxMemoryUsedForSortingExportInMB int
// Bucket used to store export.
Bucket string
// Prefix is s3 prefix where to store export inside bucket.
Prefix string
// DryRun allows to generate export and convert it to AuditEvents.
// Nothing is published to athena publisher.
// Can be used to test if export is valid.
DryRun bool
// NoOfEmitWorkers defines how many workers are used to emit audit events.
NoOfEmitWorkers int
// CheckpointPath is full path of file where checkpoint data should be stored.
// Defaults to file in current directory (athenadynamomigration.json)
// Checkpoint allow to resume export which failed during emitting.
CheckpointPath string
// TopicARN is topic of athena logger.
TopicARN string
// LargePayloadBucket is s3 bucket configured for large payloads in athena logger.
LargePayloadBucket string
// LargePayloadPrefix is s3 prefix configured for large payloads in athena logger.
LargePayloadPrefix string
Logger log.FieldLogger
// contains filtered or unexported fields
}
func (*Config) CheckAndSetDefaults ¶
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.