Skip to main content
AWSFebruary 22, 202614 min read

iOS Telemetry Pipeline with Kinesis, Glue, and Athena

Any iOS app with real users generates telemetry. Session starts, feature usage, error events, performance metrics, purchase funnels. Most teams start by shipping all of it to Amplitude or Mixpanel and calling it done. T…

Cross-posted from cloudops.consulting

Any iOS app with real users generates telemetry. Session starts, feature usage, error events, performance metrics, purchase funnels. Most teams start by shipping all of it to Amplitude or Mixpanel and calling it done. That works for a while. Then the monthly invoice triples, you discover the vendor's data model cannot answer a question your PM asked three days ago, and you realize you are paying somebody else to store your data in a format optimized for their business.

I have deployed the pipeline documented here across several production iOS applications. The app writes events straight to Amazon Data Firehose (formerly Kinesis Data Firehose) with the Amplify Firehose client, using short-lived credentials from a Cognito identity pool. Firehose delivers them reliably to S3, Glue discovers the schema automatically, and Athena gives you full SQL over the raw data. No servers. Scales from zero to billions of events per month. I provide the full infrastructure code in both Terraform and Pulumi so you can pick whichever fits your team.

Updated November 2026. The first version of this pipeline put Amazon Pinpoint in front of Firehose. AWS ended support for Pinpoint on October 30, 2026, and its migration guidance for event collection and mobile analytics is Amazon Kinesis, sent directly from the Amplify SDK. That is what this version does. Dropping the Pinpoint hop also made the pipeline simpler and cheaper.

Why a Dedicated Telemetry Pipeline

The first pushback I always hear: "Amplitude already does this." Sure. It does, at a price, with their query model, and with your data locked in their infrastructure. Once you evaluate cost, flexibility, and data ownership together, the case for owning the pipeline gets hard to argue against.

DimensionThird-Party PlatformAWS Telemetry Pipeline
Data ownershipVendor stores your data; export often limited or delayedYou own the S3 bucket and query anytime with any tool
Query flexibilityConstrained to vendor's UI and query modelFull SQL via Athena; join with any other dataset
Data residencyLimited region selection; dependent on vendor's infrastructureDeploy in any AWS region; meets sovereignty requirements
RetentionOften tiered pricing for longer retentionS3 lifecycle policies; pennies per GB-month
IntegrationAPI/webhook exports; often batched or rate-limitedDirect S3 access; feed into ML pipelines, data lakes, BI tools
Schema controlVendor's event schema with limited customizationYour schema, your structure, your partitioning

The numbers at scale speak for themselves:

Monthly Event VolumeAmplitude (Growth)Mixpanel (Growth)AWS Pipeline (estimated)
10M events~$1,000/mo~$1,100/mo~$12/mo
100M events~$4,500/mo~$5,000/mo~$67/mo
1B events~$20,000+/mo (Enterprise)~$25,000+/mo (Enterprise)~$450/mo

Those AWS estimates include Firehose ingestion, S3, Glue crawler runs, and moderate Athena query volume. At a billion events per month, this pipeline costs roughly 40x less. The gap only widens with longer retention windows because S3 storage is pennies compared to what analytics vendors charge for historical data access.

Pipeline Architecture

Standard streaming ingestion pattern. Each component does one thing:

gets temporary credentialsPutRecordBatchiOS AppAmplify Firehose clientCognitoIdentity PoolAmazon DataFirehoseAmazon S3Partitioned StorageAWS GlueSchema CrawlerGlue DataCatalogAmazonAthena
iOS telemetry pipeline architecture

I chose each component for a specific reason:

ComponentRoleScaling ModelCost Driver
Amplify Firehose clientClient-side capture, on-disk cache, batching, retriesRuns on device; flushes every 30 seconds by defaultFree (client-side)
Cognito identity poolIssues short-lived AWS credentials scoped to one Firehose streamFully managedFree
Amazon Data FirehoseReliable buffered delivery to S3Auto-scales; no shard management$0.029 per GB ingested, each record rounded up to the next 5 KB increment
Amazon S3Durable, partitioned event storageInfinite scale; pay per GB stored$0.023/GB (Standard), $0.0125/GB (IA)
AWS Glue CrawlerAutomatic schema discovery from S3 dataRuns on schedule; DPU-hour billing$0.44 per DPU-hour (billed per second)
Glue Data CatalogCentralized metadata and table definitionsManaged; free tier covers most use casesFree for first 1M objects
Amazon AthenaAd-hoc SQL queries over S3 dataServerless; per-query billing$5.00 per TB scanned

There are zero always-on compute resources here. Everything is either event-driven (Firehose) or invoked on demand (Glue crawler, Athena queries). Process zero events, pay close to nothing. Process a billion events a month, still pay a fraction of what Amplitude would charge. I have run this pipeline on a side project that generated maybe 500 events per day and the bill rounded to zero for months.

Data Flow: From Tap to Query

Follow a single telemetry event from the user tapping a button all the way to an analyst running a query. This walkthrough shows why each component exists and where the handoffs happen:

record(data, streamName)Cache on disk until the next flushGet temporary AWS credentialsCredentials for the app rolePutRecordBatch (up to 500 records)Buffer (size/time threshold)PUT object (GZIP compressed)List new partitionsUpdate table schemaGet table metadataScan partitioned dataReturn query resultsdata/year=2026/month=02/day=22/iOS AppFirehose clientCognito Identity PoolFirehoseS3 BucketGlue CrawlerGlue CatalogAthena
Telemetry event lifecycle from device to query

Step 1: Client-side event capture. The iOS app records events with AmplifyFirehoseClient, the standalone Firehose client in the Amplify Swift library (added in 2.55.0). It is not the Amplify Analytics category, which is built on Pinpoint. The client writes each record to an on-disk cache (5 MB by default), flushes every 30 seconds, retries each record up to five times, and holds events while the device is offline. Batching keeps network overhead low and avoids hammering the user's battery.

Pinpoint used to stamp every event with an application ID, session, and device context. Without it, the app owns its event envelope, which I consider an improvement because the schema is now yours:

import AmplifyFirehoseClient

struct TelemetryEvent: Encodable {
    let eventId: String
    let eventType: String
    let eventTimestamp: Int64      // epoch milliseconds, device clock
    let sessionId: String
    let appVersion: String
    let attributes: [String: String]
}

// credentialsProvider comes from Amplify Auth, backed by the Cognito identity pool
let firehose = try AmplifyFirehoseClient(
    region: "us-east-1",
    credentialsProvider: credentialsProvider)

let encoder = JSONEncoder()
encoder.keyEncodingStrategy = .convertToSnakeCase   // event_type, session_id, ...

let event = TelemetryEvent(
    eventId: UUID().uuidString,
    eventType: "checkout_started",
    eventTimestamp: Int64(Date().timeIntervalSince1970 * 1000),
    sessionId: sessionId,
    appVersion: Bundle.main.infoDictionary?["CFBundleShortVersionString"] as? String ?? "unknown",
    attributes: ["cart_items": "3"])

// One JSON object per line: Firehose adds no delimiter between records
var line = try encoder.encode(event)
line.append(0x0A)
_ = try await firehose.record(data: line, streamName: "myapp-dev-telemetry")

That trailing newline matters. Firehose concatenates records into each S3 object exactly as it receives them, and Athena's JSON reader expects one object per line.

Step 2: Direct delivery to Firehose. On each flush, the client gets temporary credentials from a Cognito identity pool and sends up to 500 cached records in one PutRecordBatch call. There is no ingestion service in between. The identity pool's role allows one action, firehose:PutRecordBatch, on one stream ARN.

Step 3: Firehose buffering and delivery. Firehose accumulates incoming records until either the buffer size (default 5 MB) or the buffer interval (default 300 seconds) trips, whichever happens first. It GZIP-compresses the batch and writes one object to S3 using a Hive-style partitioned prefix.

Step 4: S3 partitioned storage. Objects land in S3 with keys like data/year=2026/month=02/day=22/firehose-telemetry-1-2026-02-22-14-30-00-abc123.gz. This partitioning scheme is what makes Athena queries both fast and cheap. Skip it and you will regret it within a month. The partition date is when Firehose received the record, not when the user tapped the button. An event cached on a phone that was offline for a day lands in the next day's partition, so time-sensitive queries filter on event_timestamp within a partition range one day wider than they need.

Step 5: Glue schema discovery. A Glue crawler runs on schedule (every 6 hours in my configuration), picks up new partitions, infers the JSON schema from the data, and updates the Glue Data Catalog.

Step 6: Athena queries. Analysts run standard SQL through Athena. Automated reports do the same. Athena pulls table metadata from the Glue Catalog and reads directly from S3, scanning only the partitions your WHERE clause specifies.

S3 Partitioning Strategy

Partitioning is the single most impactful design decision in the entire pipeline. Full stop. Athena charges $5 per TB scanned. A query that scans 100 GB costs $0.50. Partition properly and that same query touches 1 GB: $0.005. I learned this the expensive way on an early project where I skipped partitioning to "move fast" and ended up with a $400 Athena bill in the first month.

The Firehose delivery stream uses Hive-style partitioning with this prefix pattern:

data/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/

That produces S3 keys Athena recognizes as partition columns:

s3://myapp-dev-telemetry/
├── data/
│   ├── year=2026/
│   │   ├── month=01/
│   │   │   ├── day=15/
│   │   │   │   ├── telemetry-1-2026-01-15-00-05-00-abc123.gz
│   │   │   │   ├── telemetry-1-2026-01-15-00-10-00-def456.gz
│   │   │   │   └── ...
│   │   │   ├── day=16/
│   │   │   └── ...
│   │   ├── month=02/
│   │   └── ...
│   └── ...
└── errors/
    └── ...

I tier the data with S3 lifecycle policies based on how often it gets queried:

Storage TierDays After CreationCost per GB-MonthUse Case
S3 Standard0-30$0.023Active analysis, recent events
S3 Standard-IA30-90$0.0125Historical lookback, trend analysis
Expired90+$0.000Data deleted per retention policy

All of these thresholds are configurable. In production deployments with compliance requirements, I extend expiration to 365 days or longer and add a Glacier Deep Archive tier at 180 days for audit retention.

Amazon Data Firehose Configuration

I use Firehose here instead of Kinesis Data Streams. Firehose requires no shard capacity planning, no consumer code, and no scaling logic. Data Streams gives you lower latency and consumer fan-out, but for telemetry where a few minutes of lag is fine, Firehose removes an entire category of operational work. I have operated Data Streams pipelines in other contexts and the shard management alone justified switching to Firehose for analytics use cases.

ParameterValueTradeoff
Buffer size5 MB (configurable 1-128 MB)Larger buffers = fewer S3 objects = lower S3 request costs, but higher delivery latency
Buffer interval300 seconds (configurable 0-900s)Shorter intervals = lower latency but more small files; longer intervals = better compression ratios
CompressionGZIP70-85% compression ratio for JSON telemetry; Athena reads GZIP natively
Error handlingSeparate error prefix in same bucketFailed records land in errors/ with diagnostic metadata for debugging
Firehose flushes when either the buffer size or interval threshold is reached, whichever comes first. During high-volume periods, the size threshold triggers more frequently, producing well-compressed files. During low-volume periods, the interval threshold ensures data is delivered within a predictable window. This adaptive behavior is why Firehose works well from prototype scale to production scale without reconfiguration.

In Terraform, you need the extended_s3 destination type specifically. The basic s3 destination type in older Terraform provider versions only supports a static prefix, which dumps all data into one directory and forces Athena to scan everything regardless of the time range in your query. Current versions of the AWS Terraform provider have deprecated the basic s3 destination in favor of extended_s3 exclusively. I have seen teams using the basic destination and wondering why their Athena bills are ten times what they expected.

AWS Glue: Schema Discovery and Cataloging

Glue does two things here: infer the schema and register partitions. The crawler reads sample files from S3, figures out the JSON structure, registers new date partitions as they appear, and keeps everything updated in the Glue Data Catalog. Athena uses that catalog as its metastore.

Crawler SettingValueWhy
ScheduleEvery 6 hours (cron(0 */6 * * ? *))Balances partition freshness against crawler cost
Recrawl behaviorCRAWL_NEW_FOLDERS_ONLYOnly scans new partitions; avoids re-reading historical data
Schema change policyLOG / LOGForced by CRAWL_NEW_FOLDERS_ONLY; new partitions are added, but the existing table schema is not changed
GroupingCombineCompatibleSchemasMerges slightly different schemas across partitions into a single table

Crawler schedule is a cost/freshness tradeoff:

ScheduleMonthly Cost (~)Partition Delay
Every hour~$13/moUp to 1 hour
Every 6 hours~$2.20/moUp to 6 hours
Every 12 hours~$1.10/moUp to 12 hours
Daily~$0.55/moUp to 24 hours

Six hours between data landing in S3 and being queryable in Athena is fine for most analytics work. If you absolutely need near-real-time access, add a Lambda triggered by S3 events that calls batch_create_partition to register partitions immediately. I have done this on one project where the product team wanted a live dashboard. It works, but it adds moving parts that are rarely worth it for batch analytics.

Schema evolution is nearly painless. Ship a new iOS build with additional event attributes, then run one full crawl (temporarily switch the recrawl behavior to CRAWL_EVERYTHING, then switch it back) to pick up the new columns. Older data returns NULL for those columns. No migrations, no backfills. I have gone through a dozen schema changes on a single pipeline and the only Glue change each time was that one temporary recrawl setting.

Athena: Querying Telemetry Data

Athena gives you serverless SQL directly against the S3 data. The workgroup configuration is where you enforce cost controls and keep everyone honest:

Workgroup SettingValuePurpose
Enforce configurationtruePrevents users from overriding output location or encryption
Bytes scanned cutoff1 GBKills queries that would scan more than 1 GB (cost protection)
Result encryptionSSE-KMSEncrypts query results at rest
CloudWatch metricsEnabledTracks query counts, data scanned, and execution times

That byte scan cutoff matters more than you think. One careless query without partition filters scans the entire dataset. At $5/TB, a full scan of 100 GB costs $0.50. Sounds small. Now multiply that by a team of analysts running queries all day. The 1 GB cutoff caps any single query at roughly $0.005 and trains analysts to include partition filters fast. People learn when their queries fail.

Some example queries showing how to use partitioning correctly:

-- Daily event counts for the past week (scans ~7 partitions)
SELECT year, month, day, COUNT(*) as event_count
FROM telemetry_db.data
WHERE year = '2026' AND month = '02' AND day >= '15'
GROUP BY year, month, day
ORDER BY day DESC;

-- Top event types for a specific day (scans 1 partition)
SELECT event_type, COUNT(*) as occurrences
FROM telemetry_db.data
WHERE year = '2026' AND month = '02' AND day = '22'
GROUP BY event_type
ORDER BY occurrences DESC
LIMIT 20;

-- Session duration analysis (scans 1 month of data)
-- The app sends a session_end event carrying the session's duration
SELECT
  DATE(from_unixtime(event_timestamp / 1000)) as event_date,
  AVG(CAST(attributes.duration_ms AS DOUBLE) / 1000.0) as avg_session_seconds,
  APPROX_PERCENTILE(CAST(attributes.duration_ms AS DOUBLE) / 1000.0, 0.95) as p95_session_seconds
FROM telemetry_db.data
WHERE year = '2026' AND month = '02'
  AND event_type = 'session_end'
GROUP BY DATE(from_unixtime(event_timestamp / 1000))
ORDER BY event_date;

Notice every query includes year, month, and day in the WHERE clause. Make this a habit. It keeps your Athena bill predictable and your queries fast.

IAM Architecture

Three IAM roles, each scoped to the bare minimum for its integration point. No wildcards. Each trust policy locks down to the specific AWS service, or for the app role the specific identity pool, that assumes the role:

Trust PoliciesPermissionsPutRecordBatchPutObjectGetObjectListBucketGetObjectListBucketAWSGlueServiceRoleAssumeRoleWithWebIdentityAssumeRoleAssumeRolecognito-identity.amazonaws.comApp → FirehoseRolefirehose.amazonaws.comFirehose → S3Roleglue.amazonaws.comGlue CrawlerRoleFirehoseDelivery StreamS3 TelemetryBucketGlue ServicePermissions
IAM role architecture with trust and permission relationships
RoleTrust PrincipalPermissionsScope
App → Firehose (identity pool role)cognito-identity.amazonaws.comfirehose:PutRecordBatchSpecific Firehose stream ARN only
Firehose → S3firehose.amazonaws.coms3:PutObject, s3:GetObject, s3:ListBucket, multipart upload actionsSpecific S3 bucket and objects only
Glue Crawlerglue.amazonaws.coms3:GetObject, s3:ListBucket + AWSGlueServiceRole managed policySpecific S3 bucket and objects only

The Firehose role includes an aws:SourceAccount condition in its trust policy to block confused deputy attacks. The app role is assumed through web identity federation instead, so its trust policy pins cognito-identity.amazonaws.com:aud to your identity pool ID and limits cognito-identity.amazonaws.com:amr to authenticated and unauthenticated, or to authenticated alone if you only collect telemetry from signed-in users. Both repositories expose that choice as one setting, allow_unauthenticated_identities, and keep the identity pool on the enhanced flow, whose guest session policy allows Firehose.

Treat that app role as public. Anyone who installs your app can pull its credentials off the device and send whatever they like to your stream. That is why it gets exactly one action on exactly one stream, and why nothing downstream trusts an event just because it arrived: Athena queries filter on known event types, and anything that drives billing or security comes from your backend, not from telemetry.

The Firehose role also needs s3:AbortMultipartUpload and s3:ListBucketMultipartUploads because Firehose uses multipart uploads for large buffered deliveries. I have forgotten those permissions before and spent an hour staring at cryptic access denied errors in the Firehose error log.

Cost Estimation

Cost predictability is one of the reasons I keep coming back to this architecture. Every component bills on usage with no minimums:

Component10M Events/Month100M Events/Month1B Events/Month
Cognito identity pool$0.00$0.00$0.00
Firehose (ingestion)$1.45$14.50$145.00
S3 (storage, 90-day retention)$1.50$15.00$150.00
S3 (requests)$0.50$5.00$50.00
Glue Crawler (4x daily)$2.20$2.20$2.20
Athena (moderate queries)$5.00$25.00$75.00
CloudWatch Logs$1.00$5.00$30.00
Total~$12~$67~$452

Assumptions: 3 KB average event size (JSON), one event per Firehose record, S3 storage priced on uncompressed volume as a conservative ceiling (GZIP cuts it by about 80%), Athena scanning about 1 TB, 5 TB, and 15 TB per month at the three tiers, S3 Standard for the first 30 days then Standard-IA. Firehose bills Direct PUT ingestion in 5 KB increments per record, so each 3 KB event is billed as 5 KB: 10M events are 30 GB of data but 50 GB of billable ingestion. Cognito identity pools cost nothing.

Where to cut costs further:

LeverImpactTradeoff
Pack several newline-delimited events into each Firehose recordUp to 40% lower ingestion cost at 3 KB events (five events fill 15 KB instead of five 5 KB slots)Packing logic on the client; a lost record loses several events
Increase Firehose buffer sizeFewer S3 PUT requestsHigher delivery latency
Shorten S3 retentionLinear storage cost reductionLess historical data available
Add Glacier tier instead of expiration90%+ storage savings for archive dataMinutes-to-hours retrieval time
Use columnar format (Firehose record format conversion to Parquet)50-80% Athena cost reductionNeeds a Glue table schema defined up front and a buffer of at least 64 MiB; $0.018 per GB converted, billed in 5 KB increments

Terraform vs. Pulumi: Side-by-Side

I maintain both a Terraform and a Pulumi implementation of this pipeline. They produce identical infrastructure, including the Cognito identity pool and app role; pick whichever fits your team's workflow. Each ships tests that run the whole stack against mocks (terraform test with a mocked AWS provider, and Pulumi's mock engine under pytest), so you can check a change without an AWS account. For a deeper comparison of IaC tools, see Infrastructure as Code: CloudFormation, CDK, Terraform, and Pulumi Compared.

AspectTerraformPulumi (Python)
Repositorytf-config-telemetry-pipelinepul-py-config-telemetry-pipeline
LanguageHCL (HashiCorp Configuration Language)Python
File structureOne .tf file per resource groupOne .py module per resource group
State managementTerraform state file (local or remote)Pulumi service or self-managed backend
Variable handlingvariables.tf with type constraintsPulumi.yaml config with Python helpers
Dynamic valuesString interpolation, jsonencode()Output.apply() with lambda functions
IAM policiesInline jsonencode() blocksjson.dumps() inside apply() callbacks
Total lines of code~500~550

The Firehose delivery stream is the most interesting resource to compare because it has complex nested configuration and cross-resource references:

Terraform:

resource "aws_kinesis_firehose_delivery_stream" "telemetry" {
  name        = "${local.name_prefix}-telemetry"
  destination = "extended_s3"

  extended_s3_configuration {
    role_arn   = aws_iam_role.firehose_s3.arn
    bucket_arn = aws_s3_bucket.telemetry.arn
    prefix     = "data/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/"
    error_output_prefix = "errors/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/!{firehose:error-output-type}/"

    buffering_size     = var.firehose_buffer_size_mb
    buffering_interval = var.firehose_buffer_interval_seconds
    compression_format = "GZIP"
  }
}

Pulumi (Python):

firehose = aws.kinesis.FirehoseDeliveryStream(
    "telemetry-firehose",
    name=f"{name_prefix}-telemetry",
    destination="extended_s3",
    extended_s3_configuration={
        "role_arn": firehose_s3_role.arn,
        "bucket_arn": telemetry_bucket.arn,
        "prefix": "data/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/",
        "error_output_prefix": "errors/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/!{firehose:error-output-type}/",
        "buffering_size": config["firehose_buffer_size_mb"],
        "buffering_interval": config["firehose_buffer_interval_seconds"],
        "compression_format": "GZIP",
    },
)

The structural similarity is intentional; both tools use declarative resource definitions backed by the same AWS provider. Pulumi gives you Python's full type system, IDE support, and real programming constructs for dynamic logic. Terraform's HCL is purpose-built for infrastructure and keeps things simpler for engineers who do not want to think in Python while writing infra. I tend to reach for Pulumi on projects where the infrastructure logic has conditionals and loops, and Terraform when the setup is straightforward.

Common Failure Modes

Every one of these has bitten me in production at least once. Save yourself the debugging time:

Failure ModeSymptomRoot CauseFix
Firehose delivery failuresRecords appear in errors/ prefixIAM role permissions insufficient; bucket policy conflictVerify Firehose role has s3:PutObject on the exact bucket ARN including /* suffix
Glue crawler finds no tablesEmpty database after crawler completesCrawler S3 target path does not match Firehose prefix; missing trailing slashEnsure crawler path is s3://bucket/data/ (with trailing slash) matching Firehose prefix
Athena query scans entire datasetHigh query cost; slow executionQuery missing partition filters (year, month, day in WHERE clause)Always include partition columns in WHERE; use workgroup byte cutoff as safety net
Schema mismatch after app updateNULL values in new columns; query errorsCrawler has not run since new event attributes were deployedRun crawler manually after deploying app changes; or reduce crawler interval
Small file problemThousands of tiny S3 objects per dayBuffer interval too short for the event volumeIncrease buffer interval to 900s and buffer size to 64-128 MB for high-volume streams
No events reach S3Firehose IncomingRecords stays at zero; flushes fail with AccessDeniedException (missing permission), or the credential fetch fails with NotAuthorizedException (guest access off)Identity pool role lacks firehose:PutRecordBatch on the stream ARN, or guest access is off in the identity poolGrant the action on the exact stream ARN; enable unauthenticated identities, or collect only after sign-in
Athena sees one row per fileRow counts far too low; JSON parse errorsRecords sent without a trailing newline, so each S3 object is one long lineAppend \n to every record on the device
Firehose throttling at launch spikesFlushes fail with ServiceUnavailableException; events back up on devicesA Direct PUT stream defaults to 5 MiB/second in us-east-1, us-west-2, and eu-west-1 (1 MiB/second elsewhere)Let the client retry; request a higher quota before a big launch or push campaign
Athena workgroup query failuresQueries fail immediatelyByte scan cutoff too low for the query patternIncrease bytes_scanned_cutoff_per_query; or optimize query to scan fewer partitions
Crawler DPU timeoutCrawler fails to completeToo many small files; schema too complexEnable CRAWL_NEW_FOLDERS_ONLY; use CombineCompatibleSchemas grouping

Key Architectural Recommendations

  1. Always partition by date. This is the one decision that makes or breaks the economics of the whole pipeline. Without Hive-style date partitions, every Athena query scans every byte you have ever stored.
  2. Set Firehose buffer interval to at least 300 seconds. Shorter intervals create swarms of tiny S3 objects. Each one adds GET request overhead during Athena scans. Five minutes balances latency against the small-file problem well enough for analytics.
  3. Use GZIP compression. 70-85% storage reduction. Athena reads GZIP natively. Firehose handles compression; your pipeline sees zero performance impact. There is no reason to skip this.
  4. Configure CRAWL_NEW_FOLDERS_ONLY. I cannot stress this enough. Without it, the Glue crawler re-reads every historical file on every run. On a pipeline with six months of data, that turns a $2/month crawler into a $20/month one for no benefit.
  5. Enforce the Athena workgroup configuration. Set enforce_workgroup_configuration = true. Otherwise someone will accidentally write query results to an unencrypted bucket or blow past the byte scan cutoff.
  6. Scope IAM roles to specific resource ARNs. Never Resource: "*" on the app, Firehose, or Glue roles. If an ARN exists, use it. The app role matters most because its credentials live on every user's phone. Limits the blast radius when (not if) something goes sideways.
  7. Set up lifecycle policies from day one. Telemetry accumulates faster than you expect. A 90-day retention policy with a 30-day Standard-to-IA transition keeps storage costs from surprising you in month three.
  8. Monitor the Firehose errors/ prefix. Delivery failures are completely silent unless you watch for them. Set up an S3 event notification or a CloudWatch alarm. I once lost two days of telemetry before noticing because I skipped this step on an early deployment.
  9. Start with Firehose, not Data Streams. Shard management is complexity you do not need for analytics telemetry. If sub-second latency or consumer fan-out becomes a requirement later, you can add a Data Stream upstream without redesigning anything.
  10. Codify everything. Terraform or Pulumi, I do not care which. Just do not make Firehose buffer or Glue crawler changes in the console. Manual tweaks drift, and when they cause an incident at 2 AM, nobody remembers what was changed.

Additional Resources