High-Performance CSV-to-Database Ingestion
A high-performance .NET library and CLI tool for ingesting CSV files into relational databases. Supports SQL Server, PostgreSQL, MySQL, and SQLite with automatic schema inference, intelligent indexing, bulk insert optimizations, and parallel processing capable of 100,000+ rows per second.
dotnet tool install --global Noundry.Slurp.Tool
dotnet add package Noundry.Slurp
Detects column types from CSV data samples: integers, decimals, booleans, dates, GUIDs, and strings with smart length sizing.
Automatically creates indexes based on column names (ID, email, date patterns) and data cardinality analysis.
Uses provider-specific bulk APIs: PostgreSQL COPY BINARY, SqlBulkCopy, MySqlBulkCopy, and SQLite WAL mode.
Automatic parallelism for files over 10MB. Chunks the file and processes concurrently for maximum throughput.
Rename columns during import and override inferred types with explicit SQL types.
Real-time progress events with rows processed, percentage complete, and rows-per-second throughput.
# Simplest: SQLite
slurp data.csv -p sqlite -d mydata.db
# PostgreSQL
slurp orders.csv -p postgres -s localhost -d ecommerce -u admin --password secret
# SQL Server with Windows Auth
slurp products.csv -p sqlserver -s .\SQLEXPRESS -d ProductDB
# MySQL with custom port
slurp users.csv -p mysql -s localhost --port 3307 -d userdb -u root --password root
# Custom table name and schema
slurp data.csv -p postgres -s localhost -d mydb -t imported_data --schema staging
# Drop and recreate table
slurp fresh_data.csv -p sqlite -d app.db --drop
# Skip indexes for faster initial load
slurp bulk_import.csv -p sqlserver -d analytics --no-indexes
# Semicolon-delimited European file
slurp european_data.csv -p postgres -d mydb --delimiter ";"
| Option | Alias | Description | Default |
|---|---|---|---|
| --provider | -p | Database provider | Required |
| --server | -s | Server hostname/IP | localhost |
| --database | -d | Database name (or file path for SQLite) | - |
| --table | -t | Target table name | CSV filename |
| --schema | Database schema | Provider default | |
| --username | -u | Database username | - |
| --password | Database password | - | |
| --batch-size | -b | Rows per batch for bulk insert | 10000 |
| --delimiter | CSV delimiter character | , | |
| --drop | Drop table if exists before creating | false | |
| --no-indexes | Skip automatic index creation | false | |
| --no-parallel | Disable parallel processing | false | |
| --config | -c | Configuration JSON file path | - |
| --timeout | Command timeout in seconds | 300 |
using Noundry.Slurp;
using Noundry.Slurp.Configuration;
var config = new SlurpConfiguration
{
Provider = "postgres",
Server = "localhost",
Port = 5432,
Database = "mydb",
Username = "user",
Password = "pass",
TableName = "imported_data",
BatchSize = 10000,
Parallel = true,
CreateIndexes = true
};
using var engine = new SlurpEngine(config);
var result = await engine.IngestAsync("data.csv");
Console.WriteLine($"Inserted {result.RowsInserted:N0} rows in {result.Duration.TotalSeconds:F2}s");
Console.WriteLine($"Speed: {result.RowsPerSecond:N0} rows/sec");
using var engine = new SlurpEngine(config);
engine.ProgressChanged += (sender, e) =>
{
Console.WriteLine($"Progress: {e.PercentComplete:F1}% - {e.RowsProcessed:N0}/{e.TotalRows:N0}");
};
engine.StatusChanged += (sender, message) =>
{
Console.WriteLine($"Status: {message}");
};
var result = await engine.IngestAsync("customers.csv", CancellationToken.None);
if (result.Success)
Console.WriteLine($"Table created: {result.TableCreated}, Indexes: {result.IndexesCreated}");
else
Console.WriteLine($"Error: {result.ErrorMessage}");
var config = new SlurpConfiguration
{
Provider = "sqlserver",
Server = "localhost",
Database = "ImportDB",
TableName = "products",
// Rename columns during import
ColumnMappings = new Dictionary<string, string>
{
{ "Product Name", "product_name" },
{ "Unit Price", "unit_price" },
{ "Stock Qty", "stock_quantity" }
},
// Override inferred types
ColumnTypes = new Dictionary<string, string>
{
{ "product_id", "BIGINT" },
{ "unit_price", "DECIMAL" },
{ "is_active", "BOOLEAN" }
}
};
Slurp analyzes sample rows and detects column types in priority order:
| Priority | Type | Detection Rule |
|---|---|---|
| 1 | Boolean | true/false (case-insensitive) |
| 2-4 | TinyInt/SmallInt/Int | Parses as byte/short/int range |
| 5 | BigInt | Parses as long |
| 6 | Decimal/Float | Numeric with decimal places |
| 7 | Guid | UUID format match |
| 8 | DateTime/Date | Parseable date (with/without time component) |
| 9 | Time | Time-only values |
| 10 | String | Default fallback with smart length sizing |
Slurp automatically creates indexes based on column analysis:
Column named "id" with 100% unique values is marked as PRIMARY KEY.
Columns ending with _id, _key, _code are automatically indexed.
Columns containing "email", "phone", "date", "created", "updated" get indexed.
High cardinality (>95% unique) columns get unique constraints. Low cardinality (<10%) columns get filter indexes.
Create a JSON configuration file for reusable settings. Command-line arguments override file values.
{
"provider": "postgres",
"server": "db.example.com",
"port": 5432,
"database": "production",
"username": "app_user",
"password": "${DB_PASSWORD}",
"schema": "imports",
"batchSize": 10000,
"createIndexes": true,
"dropTableIfExists": false,
"parallel": true,
"timeout": 600,
"columnMappings": {
"Original Column Name": "new_column_name"
},
"columnTypes": {
"id": "BIGINT",
"amount": "DECIMAL"
}
}
slurp data.csv -c config.json
Uses COPY BINARY protocol via NpgsqlBinaryImporter for maximum throughput.
Uses SqlBulkCopy with automatic column mappings. Supports Windows and SQL authentication.
Uses MySqlBulkCopy with configurable batch sizes and InnoDB engine.
Parameterized INSERT within transactions with WAL mode, synchronous OFF, and memory temp store for optimized writes.
| Scenario | Recommended Batch Size |
|---|---|
| Small files (<10K rows) | 1,000 - 5,000 |
| Medium files (10K-1M rows) | 10,000 (default) |
| Large files (>1M rows) | 10,000 - 25,000 |
| Network latency / Memory constrained | 5,000 |
--no-indexes), then create them separately--drop instead of manual cleanup for re-imports--timeout 600--no-parallel if row order matters| Type Name | Aliases |
|---|---|
| STRING | TEXT, VARCHAR, NVARCHAR, CHAR |
| INTEGER | INT, INT32 |
| BIGINT | INT64, LONG |
| DECIMAL | NUMERIC, MONEY |
| BOOLEAN | BIT, BOOL |
| DATETIME | TIMESTAMP, DATETIME2 |
| GUID | UNIQUEIDENTIFIER, UUID |
| JSON | JSONB |