Noundry.Slurp

High-Performance CSV-to-Database Ingestion

A high-performance .NET library and CLI tool for ingesting CSV files into relational databases. Supports SQL Server, PostgreSQL, MySQL, and SQLite with automatic schema inference, intelligent indexing, bulk insert optimizations, and parallel processing capable of 100,000+ rows per second.

dotnet tool install --global Noundry.Slurp.Tool
dotnet add package Noundry.Slurp

Key Features

Automatic Schema Inference

Detects column types from CSV data samples: integers, decimals, booleans, dates, GUIDs, and strings with smart length sizing.

Intelligent Indexing

Automatically creates indexes based on column names (ID, email, date patterns) and data cardinality analysis.

Bulk Insert Optimizations

Uses provider-specific bulk APIs: PostgreSQL COPY BINARY, SqlBulkCopy, MySqlBulkCopy, and SQLite WAL mode.

Parallel Processing

Automatic parallelism for files over 10MB. Chunks the file and processes concurrently for maximum throughput.

Column Mapping & Overrides

Rename columns during import and override inferred types with explicit SQL types.

Progress Tracking

Real-time progress events with rows processed, percentage complete, and rows-per-second throughput.

CLI Quick Start

# Simplest: SQLite
slurp data.csv -p sqlite -d mydata.db

# PostgreSQL
slurp orders.csv -p postgres -s localhost -d ecommerce -u admin --password secret

# SQL Server with Windows Auth
slurp products.csv -p sqlserver -s .\SQLEXPRESS -d ProductDB

# MySQL with custom port
slurp users.csv -p mysql -s localhost --port 3307 -d userdb -u root --password root

# Custom table name and schema
slurp data.csv -p postgres -s localhost -d mydb -t imported_data --schema staging

# Drop and recreate table
slurp fresh_data.csv -p sqlite -d app.db --drop

# Skip indexes for faster initial load
slurp bulk_import.csv -p sqlserver -d analytics --no-indexes

# Semicolon-delimited European file
slurp european_data.csv -p postgres -d mydb --delimiter ";"

CLI Options

Option Alias Description Default
--provider-pDatabase providerRequired
--server-sServer hostname/IPlocalhost
--database-dDatabase name (or file path for SQLite)-
--table-tTarget table nameCSV filename
--schemaDatabase schemaProvider default
--username-uDatabase username-
--passwordDatabase password-
--batch-size-bRows per batch for bulk insert10000
--delimiterCSV delimiter character,
--dropDrop table if exists before creatingfalse
--no-indexesSkip automatic index creationfalse
--no-parallelDisable parallel processingfalse
--config-cConfiguration JSON file path-
--timeoutCommand timeout in seconds300

Library Usage

Basic Ingestion

using Noundry.Slurp;
using Noundry.Slurp.Configuration;

var config = new SlurpConfiguration
{
 Provider = "postgres",
 Server = "localhost",
 Port = 5432,
 Database = "mydb",
 Username = "user",
 Password = "pass",
 TableName = "imported_data",
 BatchSize = 10000,
 Parallel = true,
 CreateIndexes = true
};

using var engine = new SlurpEngine(config);
var result = await engine.IngestAsync("data.csv");

Console.WriteLine($"Inserted {result.RowsInserted:N0} rows in {result.Duration.TotalSeconds:F2}s");
Console.WriteLine($"Speed: {result.RowsPerSecond:N0} rows/sec");

With Progress Tracking

using var engine = new SlurpEngine(config);

engine.ProgressChanged += (sender, e) =>
{
 Console.WriteLine($"Progress: {e.PercentComplete:F1}% - {e.RowsProcessed:N0}/{e.TotalRows:N0}");
};

engine.StatusChanged += (sender, message) =>
{
 Console.WriteLine($"Status: {message}");
};

var result = await engine.IngestAsync("customers.csv", CancellationToken.None);

if (result.Success)
 Console.WriteLine($"Table created: {result.TableCreated}, Indexes: {result.IndexesCreated}");
else
 Console.WriteLine($"Error: {result.ErrorMessage}");

Column Mappings & Type Overrides

var config = new SlurpConfiguration
{
 Provider = "sqlserver",
 Server = "localhost",
 Database = "ImportDB",
 TableName = "products",

 // Rename columns during import
 ColumnMappings = new Dictionary<string, string>
 {
 { "Product Name", "product_name" },
 { "Unit Price", "unit_price" },
 { "Stock Qty", "stock_quantity" }
 },

 // Override inferred types
 ColumnTypes = new Dictionary<string, string>
 {
 { "product_id", "BIGINT" },
 { "unit_price", "DECIMAL" },
 { "is_active", "BOOLEAN" }
 }
};

Schema Inference

Slurp analyzes sample rows and detects column types in priority order:

Priority Type Detection Rule
1Booleantrue/false (case-insensitive)
2-4TinyInt/SmallInt/IntParses as byte/short/int range
5BigIntParses as long
6Decimal/FloatNumeric with decimal places
7GuidUUID format match
8DateTime/DateParseable date (with/without time component)
9TimeTime-only values
10StringDefault fallback with smart length sizing

Intelligent Indexing

Slurp automatically creates indexes based on column analysis:

Primary Key Detection

Column named "id" with 100% unique values is marked as PRIMARY KEY.

ID/Key Column Patterns

Columns ending with _id, _key, _code are automatically indexed.

Semantic Detection

Columns containing "email", "phone", "date", "created", "updated" get indexed.

Cardinality Analysis

High cardinality (>95% unique) columns get unique constraints. Low cardinality (<10%) columns get filter indexes.

Configuration File

Create a JSON configuration file for reusable settings. Command-line arguments override file values.

{
 "provider": "postgres",
 "server": "db.example.com",
 "port": 5432,
 "database": "production",
 "username": "app_user",
 "password": "${DB_PASSWORD}",
 "schema": "imports",
 "batchSize": 10000,
 "createIndexes": true,
 "dropTableIfExists": false,
 "parallel": true,
 "timeout": 600,
 "columnMappings": {
 "Original Column Name": "new_column_name"
 },
 "columnTypes": {
 "id": "BIGINT",
 "amount": "DECIMAL"
 }
}
slurp data.csv -c config.json

Bulk Insert by Provider

PostgreSQL (Fastest)

Uses COPY BINARY protocol via NpgsqlBinaryImporter for maximum throughput.

SQL Server

Uses SqlBulkCopy with automatic column mappings. Supports Windows and SQL authentication.

MySQL

Uses MySqlBulkCopy with configurable batch sizes and InnoDB engine.

SQLite

Parameterized INSERT within transactions with WAL mode, synchronous OFF, and memory temp store for optimized writes.

Performance Tuning

Scenario Recommended Batch Size
Small files (<10K rows)1,000 - 5,000
Medium files (10K-1M rows)10,000 (default)
Large files (>1M rows)10,000 - 25,000
Network latency / Memory constrained5,000

Tips

  • Disable indexes for initial load (--no-indexes), then create them separately
  • Use --drop instead of manual cleanup for re-imports
  • Increase timeout for large files: --timeout 600
  • Use --no-parallel if row order matters
  • Use SQLite for testing -- fastest setup, no server required

Supported Type Overrides

Type Name Aliases
STRINGTEXT, VARCHAR, NVARCHAR, CHAR
INTEGERINT, INT32
BIGINTINT64, LONG
DECIMALNUMERIC, MONEY
BOOLEANBIT, BOOL
DATETIMETIMESTAMP, DATETIME2
GUIDUNIQUEIDENTIFIER, UUID
JSONJSONB