2024-08-20T05:45:24.330137Z INFO test_mlp_auto_sales::datasets::auto_sales: Completed data upload to S3
2024-08-20T05:45:24.330531Z INFO sqlx::postgres::notice: table "auto_sales_partitioned" does not exist, skipping
2024-08-20T05:45:24.330715Z INFO sqlx::postgres::notice: server "auto_sales_server" does not exist, skipping
2024-08-20T05:45:24.330874Z INFO sqlx::postgres::notice: foreign-data wrapper "parquet_wrapper" does not exist, skipping
2024-08-20T05:45:24.331018Z INFO sqlx::postgres::notice: server "auto_sales_server" does not exist, skipping
2024-08-20T05:45:25.205625Z INFO test_mlp_auto_sales::datasets::auto_sales: Starting assert_total_sales test with query:
SELECT year, manufacturer, ROUND(SUM(price)::numeric, 4)::float8 as total_sales
FROM auto_sales_partitioned
WHERE year BETWEEN 2020 AND 2024
GROUP BY year, manufacturer
ORDER BY year, total_sales DESC;
+--------------+------+--------------+---------------+
| Source | Year | Manufacturer | Total Sales |
+--------------+------+--------------+---------------+
| Pg_Analytics | 2020 | Mercedes | 11400921.4506 |
| Pg_Analytics | 2020 | Kia | 10811412.3667 |
| Pg_Analytics | 2020 | Ford | 10721787.9837 |
| Pg_Analytics | 2020 | Hyundai | 10715014.7296 |
| Pg_Analytics | 2020 | Honda | 10390930.1662 |
| Pg_Analytics | 2020 | BMW | 10381541.0703 |
| Pg_Analytics | 2020 | Toyota | 10028689.222 |
| Pg_Analytics | 2020 | Nissan | 9957042.1702 |
| Pg_Analytics | 2020 | Audi | 9602698.9083 |
| Pg_Analytics | 2020 | Chevrolet | 8774449.3195 |
| Pg_Analytics | 2021 | Kia | 11175916.2939 |
| Pg_Analytics | 2021 | Hyundai | 10642153.8687 |
| Pg_Analytics | 2021 | Toyota | 10383037.249 |
| Pg_Analytics | 2021 | Chevrolet | 10369858.9672 |
| Pg_Analytics | 2021 | Nissan | 9617196.7718 |
| Pg_Analytics | 2021 | Honda | 9547622.5708 |
| Pg_Analytics | 2021 | Ford | 9503128.5804 |
| Pg_Analytics | 2021 | Mercedes | 9325256.968 |
| Pg_Analytics | 2021 | Audi | 9056699.2692 |
| Pg_Analytics | 2021 | BMW | 8660065.3439 |
| Pg_Analytics | 2022 | Chevrolet | 10892111.3035 |
| Pg_Analytics | 2022 | Audi | 10661339.6389 |
| Pg_Analytics | 2022 | Toyota | 10522942.8592 |
| Pg_Analytics | 2022 | Mercedes | 10260270.2803 |
| Pg_Analytics | 2022 | Nissan | 10049925.9871 |
| Pg_Analytics | 2022 | BMW | 9845793.667 |
| Pg_Analytics | 2022 | Ford | 9642300.4198 |
| Pg_Analytics | 2022 | Honda | 9619521.18 |
| Pg_Analytics | 2022 | Hyundai | 9297803.3023 |
| Pg_Analytics | 2022 | Kia | 9175584.2327 |
| Pg_Analytics | 2023 | Chevrolet | 10533326.3468 |
| Pg_Analytics | 2023 | Toyota | 10530614.1897 |
| Pg_Analytics | 2023 | Honda | 10446132.503 |
| Pg_Analytics | 2023 | Hyundai | 10359928.3824 |
| Pg_Analytics | 2023 | Ford | 10175706.3077 |
| Pg_Analytics | 2023 | Kia | 10042067.0541 |
| Pg_Analytics | 2023 | Mercedes | 9495948.2358 |
| Pg_Analytics | 2023 | BMW | 9397755.3043 |
| Pg_Analytics | 2023 | Audi | 9381773.4172 |
| Pg_Analytics | 2023 | Nissan | 8829886.2025 |
| Pg_Analytics | 2024 | Hyundai | 10748091.9321 |
| Pg_Analytics | 2024 | Toyota | 10404872.6884 |
| Pg_Analytics | 2024 | Mercedes | 10275218.4575 |
| Pg_Analytics | 2024 | Ford | 10232749.5697 |
| Pg_Analytics | 2024 | Honda | 10144502.8069 |
| Pg_Analytics | 2024 | Nissan | 10029082.693 |
| Pg_Analytics | 2024 | Chevrolet | 9753982.7217 |
| Pg_Analytics | 2024 | Kia | 9593745.7302 |
| Pg_Analytics | 2024 | Audi | 9273891.787 |
| Pg_Analytics | 2024 | BMW | 7583240.7568 |
+--------------+------+--------------+---------------+
+-----------+------+--------------+---------------+
| Source | Year | Manufacturer | Total Sales |
+-----------+------+--------------+---------------+
| DataFrame | 2020 | Mercedes | 11400921.4506 |
| DataFrame | 2020 | Kia | 10811412.3667 |
| DataFrame | 2020 | Ford | 10721787.9837 |
| DataFrame | 2020 | Hyundai | 10715014.7296 |
| DataFrame | 2020 | Honda | 10390930.1662 |
| DataFrame | 2020 | BMW | 10381541.0703 |
| DataFrame | 2020 | Toyota | 10028689.222 |
| DataFrame | 2020 | Nissan | 9957042.1702 |
| DataFrame | 2020 | Audi | 9602698.9083 |
| DataFrame | 2020 | Chevrolet | 8774449.3195 |
| DataFrame | 2021 | Kia | 11175916.2939 |
| DataFrame | 2021 | Hyundai | 10642153.8687 |
| DataFrame | 2021 | Toyota | 10383037.249 |
| DataFrame | 2021 | Chevrolet | 10369858.9672 |
| DataFrame | 2021 | Nissan | 9617196.7718 |
| DataFrame | 2021 | Honda | 9547622.5708 |
| DataFrame | 2021 | Ford | 9503128.5804 |
| DataFrame | 2021 | Mercedes | 9325256.968 |
| DataFrame | 2021 | Audi | 9056699.2692 |
| DataFrame | 2021 | BMW | 8660065.3439 |
| DataFrame | 2022 | Chevrolet | 10892111.3035 |
| DataFrame | 2022 | Audi | 10661339.6389 |
| DataFrame | 2022 | Toyota | 10522942.8592 |
| DataFrame | 2022 | Mercedes | 10260270.2803 |
| DataFrame | 2022 | Nissan | 10049925.9871 |
| DataFrame | 2022 | BMW | 9845793.667 |
| DataFrame | 2022 | Ford | 9642300.4198 |
| DataFrame | 2022 | Honda | 9619521.18 |
| DataFrame | 2022 | Hyundai | 9297803.3023 |
| DataFrame | 2022 | Kia | 9175584.2327 |
| DataFrame | 2023 | Chevrolet | 10533326.3468 |
| DataFrame | 2023 | Toyota | 10530614.1897 |
| DataFrame | 2023 | Honda | 10446132.503 |
| DataFrame | 2023 | Hyundai | 10359928.3824 |
| DataFrame | 2023 | Ford | 10175706.3077 |
| DataFrame | 2023 | Kia | 10042067.0541 |
| DataFrame | 2023 | Mercedes | 9495948.2358 |
| DataFrame | 2023 | BMW | 9397755.3043 |
| DataFrame | 2023 | Audi | 9381773.4172 |
| DataFrame | 2023 | Nissan | 8829886.2025 |
| DataFrame | 2024 | Hyundai | 10748091.9321 |
| DataFrame | 2024 | Toyota | 10404872.6884 |
| DataFrame | 2024 | Mercedes | 10275218.4575 |
| DataFrame | 2024 | Ford | 10232749.5697 |
| DataFrame | 2024 | Honda | 10144502.8069 |
| DataFrame | 2024 | Nissan | 10029082.693 |
| DataFrame | 2024 | Chevrolet | 9753982.7217 |
| DataFrame | 2024 | Kia | 9593745.7302 |
| DataFrame | 2024 | Audi | 9273891.787 |
| DataFrame | 2024 | BMW | 7583240.7568 |
+-----------+------+--------------+---------------+
2024-08-20T05:45:25.959626Z INFO test_mlp_auto_sales::datasets::auto_sales: Time taken for Pg_Analytics query: 719.631507ms
2024-08-20T05:45:25.959655Z INFO test_mlp_auto_sales::datasets::auto_sales: Time taken for DataFrame operations: 31.365699ms
2024-08-20T05:45:25.959669Z INFO test_mlp_auto_sales::datasets::auto_sales: Starting assert_avg_price test with query:
SELECT manufacturer, ROUND(AVG(price)::numeric, 4)::float8 as avg_price
FROM auto_sales_partitioned
WHERE year = 2023
GROUP BY manufacturer
ORDER BY avg_price DESC;
+--------------+--------------+---------------+
| Source | Manufacturer | Average Price |
+--------------+--------------+---------------+
| Pg_Analytics | Audi | 51548.2056 |
| Pg_Analytics | Honda | 51458.7808 |
| Pg_Analytics | Hyundai | 50536.236 |
| Pg_Analytics | BMW | 50525.5662 |
| Pg_Analytics | Nissan | 49328.9732 |
| Pg_Analytics | Chevrolet | 48765.3998 |
| Pg_Analytics | Kia | 48747.8983 |
| Pg_Analytics | Mercedes | 48448.7155 |
| Pg_Analytics | Ford | 47998.6147 |
| Pg_Analytics | Toyota | 47649.838 |
+--------------+--------------+---------------+
+-----------+--------------+---------------+
| Source | Manufacturer | Average Price |
+-----------+--------------+---------------+
| DataFrame | Audi | 51548.2056 |
| DataFrame | Honda | 51458.7808 |
| DataFrame | Hyundai | 50536.236 |
| DataFrame | BMW | 50525.5662 |
| DataFrame | Nissan | 49328.9732 |
| DataFrame | Chevrolet | 48765.3998 |
| DataFrame | Kia | 48747.8983 |
| DataFrame | Mercedes | 48448.7155 |
| DataFrame | Ford | 47998.6147 |
| DataFrame | Toyota | 47649.838 |
+-----------+--------------+---------------+
2024-08-20T05:45:26.143657Z INFO test_mlp_auto_sales::datasets::auto_sales: Time taken for Pg_Analytics query: 162.759234ms
2024-08-20T05:45:26.143688Z INFO test_mlp_auto_sales::datasets::auto_sales: Time taken for DataFrame operations: 19.983599ms
2024-08-20T05:45:26.143699Z INFO test_mlp_auto_sales::datasets::auto_sales: Starting assert_monthly_sales test with query:
SELECT year, month, COUNT(*) as sales_count,
array_agg(sale_id) as sale_ids
FROM auto_sales_partitioned
WHERE manufacturer = 'Toyota' AND year = 2024
GROUP BY year, month
ORDER BY month;
+--------------+------+-------+-------------+---------------------------+
| Source | Year | Month | Sales Count | Sale IDs (first 5) |
+--------------+------+-------+-------------+---------------------------+
| Pg_Analytics | 2024 | 1 | 17 | [35, 55, 106, 240, 318] |
| Pg_Analytics | 2024 | 2 | 20 | [19, 34, 83, 95, 138] |
| Pg_Analytics | 2024 | 3 | 12 | [115, 128, 152, 174, 225] |
| Pg_Analytics | 2024 | 4 | 19 | [14, 112, 141, 156, 216] |
| Pg_Analytics | 2024 | 5 | 26 | [70, 85, 92, 94, 110] |
| Pg_Analytics | 2024 | 6 | 14 | [11, 17, 41, 105, 171] |
| Pg_Analytics | 2024 | 7 | 15 | [21, 97, 101, 192, 217] |
| Pg_Analytics | 2024 | 8 | 17 | [89, 94, 279, 378, 403] |
| Pg_Analytics | 2024 | 9 | 18 | [119, 198, 205, 341, 429] |
| Pg_Analytics | 2024 | 10 | 15 | [28, 30, 50, 76, 232] |
| Pg_Analytics | 2024 | 11 | 17 | [118, 122, 162, 166, 206] |
| Pg_Analytics | 2024 | 12 | 17 | [61, 100, 155, 161, 180] |
+--------------+------+-------+-------------+---------------------------+
+-----------+------+-------+-------------+---------------------------+
| Source | Year | Month | Sales Count | Sale IDs (first 5) |
+-----------+------+-------+-------------+---------------------------+
| DataFrame | 2024 | 1 | 17 | [35, 55, 106, 240, 318] |
| DataFrame | 2024 | 2 | 20 | [19, 34, 83, 95, 138] |
| DataFrame | 2024 | 3 | 12 | [115, 128, 152, 174, 225] |
| DataFrame | 2024 | 4 | 19 | [14, 112, 141, 156, 216] |
| DataFrame | 2024 | 5 | 26 | [70, 85, 92, 94, 110] |
| DataFrame | 2024 | 6 | 14 | [11, 17, 41, 105, 171] |
| DataFrame | 2024 | 7 | 15 | [21, 97, 101, 192, 217] |
| DataFrame | 2024 | 8 | 17 | [89, 94, 279, 378, 403] |
| DataFrame | 2024 | 9 | 18 | [119, 198, 205, 341, 429] |
| DataFrame | 2024 | 10 | 15 | [28, 30, 50, 76, 232] |
| DataFrame | 2024 | 11 | 17 | [118, 122, 162, 166, 206] |
| DataFrame | 2024 | 12 | 17 | [61, 100, 155, 161, 180] |
+-----------+------+-------+-------------+---------------------------+
2024-08-20T05:45:26.183418Z INFO test_mlp_auto_sales::datasets::auto_sales: Time taken for Pg_Analytics query: 18.300859ms
2024-08-20T05:45:26.183448Z INFO test_mlp_auto_sales::datasets::auto_sales: Time taken for DataFrame operations: 19.868635ms
test test_partitioned_automotive_sales_s3_parquet ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.34s
Created
August 20, 2024 05:46
-
-
Save shamb0/2ed909ac9604c610af1d7fa0e87f9a82 to your computer and use it in GitHub Desktop.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Super clean :)