Step 03
Cleaning the data
Raw data is almost never ready to use. Before analyzing anything, I looked for problems that could quietly make the numbers wrong. This card from the dashboard counts what I found:
Data checks
found before analysis- Orders not delivered2,963
- Repeat buyers hidden by customer ID3,345
- Orders without any item775
- Products without a category610
- Orders with 2+ reviews547
- Payment ≠ items + freight249
- Delivered, but no delivery date8
And this is what I decided to do about each one:
ProblemMy decision
2,963 orders were never delivered (shipped, canceled, unavailable…)Left out of revenue and delivery numbers
8 delivered orders have no delivery dateLeft out, since delivery time can't be calculated
775 orders have no itemsThey drop out on their own when I join the items table
547 orders have more than one reviewKept only the latest review per order
A new customer_id is created for every orderUsed customer_unique_id to recognize the same person. Otherwise, 3,345 repeat buyers disappear.
610 products have no categoryLabeled them “unknown” so they still count
249 orders where payments don't match items + freight (by more than R$1)Calculated revenue from item prices, and only used payments for the payment mix
2016 and late 2018 are incompleteMonthly trends only cover Jan 2017 to Aug 2018
What counts as “late”?Arriving after the promised date. Arriving on the day itself isn't late.
I wrote each decision once, as a SQL view, so all 11 queries use the exact same clean data. Here's the main one:
-- Only orders that really reached the customer.
-- Everything about revenue and delivery is based on this view.
CREATE VIEW delivered_orders AS
SELECT o.order_id,
c.customer_unique_id, -- the real person (customer_id changes per order)
c.customer_state,
strftime('%Y-%m', o.order_purchase_timestamp) AS purchase_month,
julianday(o.order_delivered_customer_date)
- julianday(o.order_purchase_timestamp) AS delivery_days,
-- arriving on the promised day is NOT late
CASE WHEN date(o.order_delivered_customer_date)
> date(o.order_estimated_delivery_date)
THEN 1 ELSE 0 END AS is_late
FROM orders o
JOIN customers c ON c.customer_id = o.customer_id
WHERE o.order_status = 'delivered'
AND o.order_delivered_customer_date IS NOT NULL;